Jul 14 14:30:10.438246 ip-10-0-141-194 systemd[1]: Starting Kubernetes Kubelet... Jul 14 14:30:10.903806 ip-10-0-141-194 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 14:30:10.903806 ip-10-0-141-194 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 14:30:10.903806 ip-10-0-141-194 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 14:30:10.904403 ip-10-0-141-194 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 14:30:10.904403 ip-10-0-141-194 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 14:30:10.906323 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.906229 2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 14:30:10.912948 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912930 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 14:30:10.912948 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912948 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912953 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912957 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912970 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912973 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912975 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912978 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912981 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912983 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912986 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912989 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912991 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912993 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.912996 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913000 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913005 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913009 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913012 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913019 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 14:30:10.913026 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913023 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913025 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913029 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913031 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913035 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913037 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913040 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913043 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913046 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913048 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913051 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913053 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913056 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913058 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913061 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913064 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913067 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913069 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913072 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 14:30:10.913440 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913074 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913077 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913080 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913082 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913084 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913087 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913091 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913094 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913097 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913099 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913102 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913105 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913107 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913110 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913113 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913116 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913119 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913122 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913125 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 14:30:10.913869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913127 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913130 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913132 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913134 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913137 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913139 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913142 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913144 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913147 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913149 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913152 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913155 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913158 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913161 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913163 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913166 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913168 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913170 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913173 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913175 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 14:30:10.914349 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913178 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913180 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913183 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913185 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913187 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913190 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913192 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913195 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913198 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913201 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913204 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913206 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913209 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913212 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913214 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913217 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913220 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913222 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913225 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913227 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 14:30:10.914861 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.913230 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 14:30:10.915713 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915699 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 14:30:10.915713 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915712 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915716 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915719 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915722 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915725 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915728 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915731 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915734 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915736 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915739 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915741 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915744 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915747 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915749 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915751 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915754 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915757 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915759 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915762 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915764 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 14:30:10.915766 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915767 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915769 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915772 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915775 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915778 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915780 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915782 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915785 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915787 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915790 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915793 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915795 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915797 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915800 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915802 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915805 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915807 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915810 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915812 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915815 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 14:30:10.916277 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915819 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915823 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915825 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915828 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915831 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915833 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915837 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915841 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915844 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915847 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915850 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915853 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915855 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915858 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915862 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915865 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915868 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915870 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915874 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 14:30:10.916729 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915877 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915880 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915883 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915885 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915888 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915890 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915893 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915896 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915898 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915901 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915903 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915906 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915909 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915912 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915914 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915916 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915919 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915921 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915923 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915926 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 14:30:10.917183 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915928 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915931 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915933 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915935 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915938 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915940 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915943 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915946 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915949 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915951 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915954 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915956 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915982 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915986 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915990 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915993 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915995 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.915998 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916000 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916392 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 14:30:10.917636 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916397 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916399 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916402 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916405 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916408 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916410 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916415 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916418 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916421 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916423 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916426 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916428 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916430 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916433 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916436 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916438 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916441 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916443 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916446 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916449 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 14:30:10.918193 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916452 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916454 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916456 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916459 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916461 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916464 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916466 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916468 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916471 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916473 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916476 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916478 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916481 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916483 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916485 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916488 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916491 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916493 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916495 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916498 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 14:30:10.918687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916500 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916503 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916505 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916508 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916511 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916513 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916516 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916518 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916520 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916522 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916526 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916528 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916531 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916533 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916535 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916538 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916540 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916543 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916545 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 14:30:10.919202 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916549 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916553 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916556 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916559 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916561 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916564 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916566 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916568 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916571 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916574 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916577 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916580 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916582 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916585 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916587 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916589 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916594 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916598 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916602 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 14:30:10.919665 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916605 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916608 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916611 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916614 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916617 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916619 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916622 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916625 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916628 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916630 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916633 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916635 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916638 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916640 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916643 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916645 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916647 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916650 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916653 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.916655 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 14:30:10.920117 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916739 2550 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916747 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916756 2550 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916764 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916770 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916774 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916778 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916782 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916785 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916789 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916793 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916796 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916799 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916802 2550 flags.go:64] FLAG: --cgroup-root="" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916806 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916808 2550 flags.go:64] FLAG: --client-ca-file="" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916812 2550 flags.go:64] FLAG: --cloud-provider="external" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916815 2550 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916819 2550 flags.go:64] FLAG: --cluster-domain="" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916822 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916825 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916828 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916831 2550 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916835 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 14:30:10.920575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916838 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916841 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916844 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916848 2550 flags.go:64] FLAG: --contention-profiling="false" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916850 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916853 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916856 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916859 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916863 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916866 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916869 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916872 2550 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916875 2550 flags.go:64] FLAG: --enable-server="true" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916878 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916882 2550 flags.go:64] FLAG: --event-burst="100" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916885 2550 flags.go:64] FLAG: --event-qps="50" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916888 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916891 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916895 2550 flags.go:64] FLAG: --eviction-hard="" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916898 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916901 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916904 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916907 2550 flags.go:64] FLAG: --eviction-soft="" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916910 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916913 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 14:30:10.921127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916915 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916918 2550 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916921 2550 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916924 2550 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916926 2550 flags.go:64] FLAG: --feature-gates="" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916930 2550 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916933 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916936 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916939 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916942 2550 flags.go:64] FLAG: --healthz-port="10248" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916945 2550 flags.go:64] FLAG: --help="false" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916947 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-141-194.ec2.internal" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916950 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916953 2550 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916956 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916970 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916974 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916977 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916980 2550 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916983 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916986 2550 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916989 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916992 2550 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916995 2550 flags.go:64] FLAG: --kube-reserved="" Jul 14 14:30:10.921726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.916998 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917001 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917004 2550 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917007 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917009 2550 flags.go:64] FLAG: --lock-file="" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917012 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917015 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917018 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917023 2550 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917025 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917028 2550 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917031 2550 flags.go:64] FLAG: --logging-format="text" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917034 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917037 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917040 2550 flags.go:64] FLAG: --manifest-url="" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917042 2550 flags.go:64] FLAG: --manifest-url-header="" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917047 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917050 2550 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917054 2550 flags.go:64] FLAG: --max-pods="110" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917057 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917060 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917062 2550 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917065 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917068 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917071 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 14:30:10.922430 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917076 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917084 2550 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917087 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917089 2550 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917092 2550 flags.go:64] FLAG: --pod-cidr="" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917099 2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917104 2550 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917107 2550 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917110 2550 flags.go:64] FLAG: --pods-per-core="0" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917113 2550 flags.go:64] FLAG: --port="10250" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917115 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917118 2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-007259f744c10f628" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917121 2550 flags.go:64] FLAG: --qos-reserved="" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917124 2550 flags.go:64] FLAG: --read-only-port="10255" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917127 2550 flags.go:64] FLAG: --register-node="true" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917130 2550 flags.go:64] FLAG: --register-with-taints="" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917134 2550 flags.go:64] FLAG: --registry-burst="10" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917136 2550 flags.go:64] FLAG: --registry-qps="5" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917139 2550 flags.go:64] FLAG: --reserved-cpus="" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917142 2550 flags.go:64] FLAG: --reserved-memory="" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917145 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917149 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917152 2550 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917156 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917159 2550 flags.go:64] FLAG: --runonce="false" Jul 14 14:30:10.923049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917161 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917164 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917168 2550 flags.go:64] FLAG: --seccomp-default="false" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917171 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917173 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917176 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917179 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917182 2550 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917187 2550 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917190 2550 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917192 2550 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917195 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917198 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917202 2550 flags.go:64] FLAG: --system-cgroups="" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917205 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917211 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917213 2550 flags.go:64] FLAG: --tls-cert-file="" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917216 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917220 2550 flags.go:64] FLAG: --tls-min-version="" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917223 2550 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917226 2550 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917229 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917231 2550 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917234 2550 flags.go:64] FLAG: --v="2" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917238 2550 flags.go:64] FLAG: --version="false" Jul 14 14:30:10.923795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917242 2550 flags.go:64] FLAG: --vmodule="" Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917245 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.917249 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917364 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917367 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917371 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917374 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917376 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917379 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917381 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917384 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917386 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917389 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917391 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917394 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917398 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917402 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917405 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917408 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 14:30:10.924408 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917411 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917413 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917416 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917419 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917422 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917424 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917427 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917431 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917434 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917437 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917439 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917442 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917444 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917447 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917449 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917452 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917455 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917457 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917461 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 14:30:10.924841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917464 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917466 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917469 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917471 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917474 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917476 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917478 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917481 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917483 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917488 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917491 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917493 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917496 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917499 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917501 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917504 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917506 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917510 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917514 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917517 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 14:30:10.925298 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917520 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917523 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917525 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917528 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917531 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917533 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917536 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917538 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917541 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917543 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917545 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917549 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917552 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917554 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917557 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917559 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917562 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917564 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917567 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917569 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 14:30:10.925755 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917571 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917576 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917578 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917581 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917583 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917586 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917588 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917591 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917594 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917597 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917599 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917602 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917604 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917606 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917609 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917611 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917614 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917616 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917618 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917621 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 14:30:10.926275 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917623 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917625 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917628 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.917632 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.918511 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.926135 2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.926154 2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.926174 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.926179 2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926244 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926249 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926252 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926255 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926258 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 14:30:10.926775 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926260 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926263 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926266 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926269 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926271 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926274 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926277 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926285 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926288 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926291 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926293 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926301 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926304 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926307 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926309 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926312 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926314 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926317 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926319 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926323 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 14:30:10.927131 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926327 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926329 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926332 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926341 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926344 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926347 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926350 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926353 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926367 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926370 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926373 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926376 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926380 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926383 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926386 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926390 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926412 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926415 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926418 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 14:30:10.927611 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926421 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926430 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926433 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926435 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926438 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926440 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926443 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926445 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926448 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926450 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926453 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926455 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926458 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926460 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926463 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926465 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926468 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926472 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926474 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926477 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 14:30:10.928077 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926479 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926482 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926484 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926487 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926489 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926492 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926494 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926497 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926500 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926502 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926505 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926507 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926510 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926512 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926520 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926523 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926525 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926528 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926530 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926533 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 14:30:10.928538 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926536 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926538 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926540 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926543 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926546 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926548 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926551 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926553 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926556 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926559 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926562 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926564 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926567 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926569 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926572 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 14:30:10.929045 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.926577 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926735 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926740 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926743 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926746 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926749 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926752 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926756 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926759 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926762 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926765 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926768 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926777 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926780 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926783 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926785 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926788 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926791 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926794 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 14:30:10.929414 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926796 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926798 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926801 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926804 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926806 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926808 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926811 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926814 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926817 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926819 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926822 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926825 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926829 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926831 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926834 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926837 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926839 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926842 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926844 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 14:30:10.929856 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926847 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926849 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926852 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926854 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926857 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926859 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926862 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926871 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926874 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926877 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926879 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926882 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926884 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926887 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926889 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926892 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926894 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926896 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926899 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926901 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 14:30:10.930381 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926904 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926907 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926909 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926912 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926914 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926917 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926919 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926922 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926924 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926926 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926929 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926932 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926934 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926937 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926939 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926942 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926944 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926947 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926949 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926951 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 14:30:10.930869 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926973 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926977 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926980 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926982 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926985 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926988 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926990 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926992 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926995 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.926997 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927000 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927002 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927005 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927007 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927010 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927013 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927015 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927019 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927022 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 14:30:10.931356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927025 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 14:30:10.931809 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927028 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 14:30:10.931809 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:10.927031 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 14:30:10.931809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.927036 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 14:30:10.931809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.927184 2550 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 14:30:10.931809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.929426 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 14:30:10.931809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.931027 2550 server.go:1019] "Starting client certificate rotation" Jul 14 14:30:10.931809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.931135 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 14:30:10.931809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.931175 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 14:30:10.934092 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.934076 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 14:30:10.939631 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.939608 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 14:30:10.950024 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.950001 2550 log.go:25] "Validated CRI v1 runtime API" Jul 14 14:30:10.952648 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.952633 2550 log.go:25] "Validated CRI v1 image API" Jul 14 14:30:10.953433 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.953415 2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 14:30:10.956053 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.956035 2550 fs.go:135] Filesystem UUIDs: map[739665c4-b89b-4f5f-8afb-9c453a27d5a7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f01147cd-fae0-441a-943a-dc8d3f1346fe:/dev/nvme0n1p4] Jul 14 14:30:10.956100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.956053 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 14:30:10.961792 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.961645 2550 manager.go:216] Machine: {Timestamp:2026-07-14 14:30:10.960241852 +0000 UTC m=+0.395226474 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3152127 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec244a719083f648307f509b446e45cd SystemUUID:ec244a71-9083-f648-307f-509b446e45cd BootID:624da748-c473-4640-8501-ef8e2584b2e4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2e:8b:95:77:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2e:8b:95:77:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:21:89:02:3a:99 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 14:30:10.961792 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.961784 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 14:30:10.961910 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.961879 2550 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 14:30:10.965336 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.965293 2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 14:30:10.965510 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.965347 2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 14:30:10.966890 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.966860 2550 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 14:30:10.966930 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.966894 2550 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 14:30:10.966930 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.966901 2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 14:30:10.967348 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.967330 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 14:30:10.967741 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.967729 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 14:30:10.969509 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.969497 2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 14:30:10.973859 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.973846 2550 state_mem.go:36] "Initialized new in-memory state store" Jul 14 14:30:10.973999 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.973989 2550 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 14:30:10.974031 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.974021 2550 kubelet.go:486] "Attempting to sync node with API server" Jul 14 14:30:10.974058 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.974038 2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 14:30:10.974058 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.974057 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 14:30:10.974117 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.974070 2550 kubelet.go:398] "Adding apiserver pod source" Jul 14 14:30:10.974117 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.974082 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 14:30:10.974541 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.974510 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 14:30:10.974541 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.974542 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 14:30:10.977135 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.977117 2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 14:30:10.979877 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.979861 2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 14:30:10.979956 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.979898 2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 14:30:10.980106 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980093 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 14:30:10.980163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980113 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 14:30:10.980163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980122 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 14:30:10.980163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980130 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 14:30:10.980163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980142 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 14:30:10.980163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980150 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 14:30:10.980983 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980972 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 14:30:10.981040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980985 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 14:30:10.981040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.980998 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 14:30:10.981040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.981006 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 14:30:10.981040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.981018 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 14:30:10.981040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.981030 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 14:30:10.981911 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.981901 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 14:30:10.981977 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.981913 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 14:30:10.986078 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.986051 2550 server.go:1268] "Started kubelet" Jul 14 14:30:10.986229 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.986198 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 14:30:10.986286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.986258 2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 14:30:10.986878 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.986856 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 14:30:10.986994 ip-10-0-141-194 systemd[1]: Started Kubernetes Kubelet. Jul 14 14:30:10.987594 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.987573 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 14:30:10.987682 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:10.987613 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 14:30:10.987848 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.987834 2550 server.go:310] "Adding debug handlers to kubelet server" Jul 14 14:30:10.987938 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:10.987916 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 14:30:10.988394 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.988379 2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 14:30:10.994506 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:10.993264 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-194.ec2.internal.18c22e645d1ab599 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-194.ec2.internal,UID:ip-10-0-141-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-194.ec2.internal,},FirstTimestamp:2026-07-14 14:30:10.985776537 +0000 UTC m=+0.420761151,LastTimestamp:2026-07-14 14:30:10.985776537 +0000 UTC m=+0.420761151,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-194.ec2.internal,}" Jul 14 14:30:10.995078 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:10.995062 2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 14:30:10.996767 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.996752 2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 14:30:10.996832 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.996759 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 14:30:10.996882 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:10.996837 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:10.997591 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.997572 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 14:30:10.997669 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.997592 2550 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 14:30:10.997669 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.997607 2550 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 14:30:10.997759 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.997686 2550 reconstruct.go:97] "Volume reconstruction finished" Jul 14 14:30:10.997759 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.997694 2550 reconciler.go:29] "Reconciler: start to sync state" Jul 14 14:30:10.997879 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.997862 2550 factory.go:55] Registering systemd factory Jul 14 14:30:10.997927 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.997884 2550 factory.go:223] Registration of the systemd container factory successfully Jul 14 14:30:10.998130 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.998114 2550 factory.go:153] Registering CRI-O factory Jul 14 14:30:10.998130 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.998131 2550 factory.go:223] Registration of the crio container factory successfully Jul 14 14:30:10.998259 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.998180 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 14:30:10.998259 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.998203 2550 factory.go:103] Registering Raw factory Jul 14 14:30:10.998259 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.998219 2550 manager.go:1196] Started watching for new ooms in manager Jul 14 14:30:10.998631 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:10.998617 2550 manager.go:349] Starting recovery of all containers Jul 14 14:30:11.008418 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.008355 2550 manager.go:354] Recovery completed Jul 14 14:30:11.010492 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.010452 2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 14:30:11.010492 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.010469 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 14:30:11.013630 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.013618 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 14:30:11.015676 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.015648 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientMemory" Jul 14 14:30:11.015775 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.015701 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 14:30:11.015775 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.015715 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientPID" Jul 14 14:30:11.016281 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.016264 2550 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 14:30:11.016281 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.016278 2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 14:30:11.016419 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.016268 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 14:30:11.016419 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.016296 2550 state_mem.go:36] "Initialized new in-memory state store" Jul 14 14:30:11.017199 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.017185 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientMemory" Jul 14 14:30:11.017263 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.017213 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 14:30:11.017263 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.017226 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientPID" Jul 14 14:30:11.018284 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.018270 2550 policy_none.go:49] "None policy: Start" Jul 14 14:30:11.018332 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.018289 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 14:30:11.018332 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.018303 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 14:30:11.021162 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.021149 2550 policy_none.go:47] "Start" Jul 14 14:30:11.025390 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.025314 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-194.ec2.internal.18c22e645ee2f397 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-194.ec2.internal,UID:ip-10-0-141-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-194.ec2.internal,},FirstTimestamp:2026-07-14 14:30:11.015676823 +0000 UTC m=+0.450661440,LastTimestamp:2026-07-14 14:30:11.015676823 +0000 UTC m=+0.450661440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-194.ec2.internal,}" Jul 14 14:30:11.027739 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.027662 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-194.ec2.internal.18c22e645ee36d65 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-194.ec2.internal,UID:ip-10-0-141-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-194.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-194.ec2.internal,},FirstTimestamp:2026-07-14 14:30:11.015708005 +0000 UTC m=+0.450692620,LastTimestamp:2026-07-14 14:30:11.015708005 +0000 UTC m=+0.450692620,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-194.ec2.internal,}" Jul 14 14:30:11.029772 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.029754 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cv9ft" Jul 14 14:30:11.037875 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.037782 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-194.ec2.internal.18c22e645ee39c74 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-194.ec2.internal,UID:ip-10-0-141-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-194.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-194.ec2.internal,},FirstTimestamp:2026-07-14 14:30:11.015720052 +0000 UTC m=+0.450704670,LastTimestamp:2026-07-14 14:30:11.015720052 +0000 UTC m=+0.450704670,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-194.ec2.internal,}" Jul 14 14:30:11.039259 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.039238 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cv9ft" Jul 14 14:30:11.057595 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.057580 2550 manager.go:337] "Starting Device Plugin manager" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.057613 2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.057623 2550 server.go:87] "Starting device plugin registration server" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.057898 2550 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.057917 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.058063 2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.058136 2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.058144 2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.058678 2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 14:30:11.079163 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.058729 2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.112391 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.112337 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 14:30:11.113524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.113507 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 14:30:11.113589 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.113545 2550 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 14:30:11.113589 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.113569 2550 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 14:30:11.113682 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.113603 2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 14:30:11.116791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.116774 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 14:30:11.159045 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.158988 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 14:30:11.160078 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.160039 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientMemory" Jul 14 14:30:11.160209 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.160101 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 14:30:11.160209 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.160115 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientPID" Jul 14 14:30:11.160209 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.160137 2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.173942 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.173918 2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.173942 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.173944 2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-194.ec2.internal\": node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.188258 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.188230 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.214835 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.214664 2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal"] Jul 14 14:30:11.214989 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.214895 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 14:30:11.215947 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.215923 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientMemory" Jul 14 14:30:11.216023 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.215956 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 14:30:11.216023 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.215985 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientPID" Jul 14 14:30:11.218331 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.218319 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 14:30:11.218470 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.218443 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.218516 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.218485 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 14:30:11.219096 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.219077 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientMemory" Jul 14 14:30:11.219096 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.219086 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientMemory" Jul 14 14:30:11.219261 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.219108 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 14:30:11.219261 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.219108 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 14:30:11.219261 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.219131 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientPID" Jul 14 14:30:11.219261 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.219119 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientPID" Jul 14 14:30:11.221311 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.221292 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.221389 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.221330 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 14:30:11.222043 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.222025 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientMemory" Jul 14 14:30:11.222101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.222055 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 14:30:11.222101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.222069 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeHasSufficientPID" Jul 14 14:30:11.252715 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.252695 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-194.ec2.internal\" not found" node="ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.256942 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.256928 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-194.ec2.internal\" not found" node="ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.288854 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.288792 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.298567 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.298541 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1acd846e89bc70267fc722d6edc8fc43-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.ec2.internal\" (UID: \"1acd846e89bc70267fc722d6edc8fc43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.298652 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.298572 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b675f1bf43f9d3e7b9ee8182f17d78ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal\" (UID: \"b675f1bf43f9d3e7b9ee8182f17d78ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.298652 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.298596 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b675f1bf43f9d3e7b9ee8182f17d78ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal\" (UID: \"b675f1bf43f9d3e7b9ee8182f17d78ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.389732 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.389698 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.399046 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.399013 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1acd846e89bc70267fc722d6edc8fc43-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.ec2.internal\" (UID: \"1acd846e89bc70267fc722d6edc8fc43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.399046 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.399030 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1acd846e89bc70267fc722d6edc8fc43-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.ec2.internal\" (UID: \"1acd846e89bc70267fc722d6edc8fc43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.399205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.399071 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b675f1bf43f9d3e7b9ee8182f17d78ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal\" (UID: \"b675f1bf43f9d3e7b9ee8182f17d78ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.399205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.399092 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b675f1bf43f9d3e7b9ee8182f17d78ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal\" (UID: \"b675f1bf43f9d3e7b9ee8182f17d78ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.399205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.399110 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b675f1bf43f9d3e7b9ee8182f17d78ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal\" (UID: \"b675f1bf43f9d3e7b9ee8182f17d78ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.399205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.399124 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b675f1bf43f9d3e7b9ee8182f17d78ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal\" (UID: \"b675f1bf43f9d3e7b9ee8182f17d78ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.490399 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.490365 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.554914 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.554840 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.559898 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.559585 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.590724 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.590689 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.697083 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.697051 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.798021 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.797978 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.898414 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:11.898334 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-194.ec2.internal\" not found" Jul 14 14:30:11.901777 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.901759 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 14:30:11.931217 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.931181 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 14:30:11.931717 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.931395 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 14:30:11.931717 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.931401 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 14:30:11.975247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.975197 2550 apiserver.go:52] "Watching apiserver" Jul 14 14:30:11.995521 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.995488 2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 14:30:11.997103 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.996538 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wrmgv","kube-system/konnectivity-agent-ppp5w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq","openshift-image-registry/node-ca-mrt4j","openshift-multus/multus-additional-cni-plugins-dkvx6","openshift-network-diagnostics/network-check-target-m9xxq","openshift-cluster-node-tuning-operator/tuned-bl9f7","openshift-dns/node-resolver-w4pnw","openshift-multus/multus-j5vxr","openshift-multus/network-metrics-daemon-r5pzf","openshift-network-operator/iptables-alerter-dgqsn","openshift-ovn-kubernetes/ovnkube-node-6998d"] Jul 14 14:30:11.997591 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.997567 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" Jul 14 14:30:11.997660 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.997577 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 14:30:11.999491 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:11.999472 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001710 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001685 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-modprobe-d\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001788 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001723 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-lib-modules\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001788 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001740 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.001788 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001750 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmw52\" (UniqueName: \"kubernetes.io/projected/430dfc42-dfe5-4568-b1bf-4afada865d71-kube-api-access-tmw52\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001788 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001775 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysconfig\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001934 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001823 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysctl-d\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001934 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001857 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysctl-conf\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001934 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001880 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-tuned\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001934 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001906 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-kubernetes\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.001934 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001931 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-systemd\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.002183 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001957 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-sys\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.002183 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.001996 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-host\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.002183 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.002020 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-run\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.002183 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.002044 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-var-lib-kubelet\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.002183 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.002087 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/430dfc42-dfe5-4568-b1bf-4afada865d71-tmp\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.002356 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.002242 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 14:30:12.002356 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.002242 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 14:30:12.002451 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.002422 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wg7cq\"" Jul 14 14:30:12.004184 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.004156 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 14:30:12.004296 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.004278 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.004356 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.004342 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 14:30:12.004806 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.004788 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 14:30:12.004883 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.004789 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 14:30:12.004883 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.004789 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-blvss\"" Jul 14 14:30:12.006614 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.006600 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.006679 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.006665 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:12.008913 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.008895 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:12.009547 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.009523 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 14:30:12.009604 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.009556 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 14:30:12.009655 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.009607 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 14:30:12.009655 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.009615 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 14:30:12.009655 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.009616 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 14:30:12.009655 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.009651 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rv8xm\"" Jul 14 14:30:12.009823 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.009657 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 14:30:12.011329 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.011314 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.011693 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.011678 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wpl8p\"" Jul 14 14:30:12.011853 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.011838 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 14:30:12.012180 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.012164 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 14:30:12.013636 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.013618 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.013950 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.013929 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 14:30:12.014209 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.014189 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 14:30:12.014426 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.014410 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7bn78\"" Jul 14 14:30:12.014680 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.014666 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 14:30:12.015733 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.015717 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 14:30:12.016160 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.016142 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.016312 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.016287 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 14:30:12.016398 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.016374 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" Jul 14 14:30:12.016459 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.016437 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 14:30:12.016532 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.016500 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 14:30:12.016572 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.016542 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pdgz2\"" Jul 14 14:30:12.016885 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.016871 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 14:30:12.018611 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.018598 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:12.018679 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.018663 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:12.019137 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.019119 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 14:30:12.019205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.019172 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 14:30:12.019255 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.019216 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8jbh8\"" Jul 14 14:30:12.032723 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.032695 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.034138 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.033844 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 14:30:12.036147 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.036031 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 14:30:12.036400 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.036353 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:12.036509 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.036418 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 14:30:12.036567 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.036537 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:12.036652 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.036632 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c45lb\"" Jul 14 14:30:12.038172 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.038150 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal"] Jul 14 14:30:12.038275 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.038194 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal"] Jul 14 14:30:12.038325 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.038288 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.040850 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.040831 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 14:30:12.040850 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.040842 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 14:30:12.041063 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.040868 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-htgkv\"" Jul 14 14:30:12.041063 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.040904 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 14:30:12.046391 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.046325 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 14:25:11 +0000 UTC" deadline="2028-01-10 17:00:10.014100612 +0000 UTC" Jul 14 14:30:12.046473 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.046397 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13082h29m57.967715844s" Jul 14 14:30:12.086118 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.086095 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-psls4" Jul 14 14:30:12.094729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.094707 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-psls4" Jul 14 14:30:12.098929 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.098905 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 14:30:12.102707 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102681 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovnkube-script-lib\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.102808 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102718 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-node-log\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.102808 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102744 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5e9aa45-eaf8-4317-b370-780af8488466-host-slash\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.102808 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102783 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-cni-bin\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.102921 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102822 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-conf-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.102921 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102859 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-device-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.102921 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102893 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-cni-binary-copy\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.103074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102921 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:12.103074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102946 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5e9aa45-eaf8-4317-b370-780af8488466-iptables-alerter-script\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.103074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102962 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-cnibin\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.103074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.102997 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-etc-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.103074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103023 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-cni-bin\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.103074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103038 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c078bb29-1f4d-45f5-beed-4d91e9a4a288-hosts-file\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.103074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103053 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c078bb29-1f4d-45f5-beed-4d91e9a4a288-tmp-dir\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.103286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103085 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysctl-conf\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.103286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103113 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-tuned\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.103286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103175 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-socket-dir-parent\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.103286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103214 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-netns\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.103286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103239 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-kubelet\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.103286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103274 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-etc-kubernetes\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.103524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103288 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysctl-conf\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.103524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103302 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.103524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103334 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-sys\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.103524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103383 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-env-overrides\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.103524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103463 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-cni-netd\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.103524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103450 2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 14:30:12.103524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103467 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-sys\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.103524 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103499 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103526 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103549 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-cnibin\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103573 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-os-release\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103617 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/430dfc42-dfe5-4568-b1bf-4afada865d71-tmp\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103652 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-os-release\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103678 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25f418cb-ca33-4715-8acf-7a84b44371cc-cni-binary-copy\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103710 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-host\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103738 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-serviceca\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.103776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103765 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-system-cni-dir\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103789 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdh9\" (UniqueName: \"kubernetes.io/projected/f5e9aa45-eaf8-4317-b370-780af8488466-kube-api-access-8hdh9\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103814 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-modprobe-d\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103834 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-lib-modules\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103849 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klbg\" (UniqueName: \"kubernetes.io/projected/25f418cb-ca33-4715-8acf-7a84b44371cc-kube-api-access-5klbg\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103865 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-ovn\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103919 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-log-socket\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.103958 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovn-node-metrics-cert\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104007 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csngt\" (UniqueName: \"kubernetes.io/projected/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-kube-api-access-csngt\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104028 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-lib-modules\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104024 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-modprobe-d\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104031 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-registration-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104078 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-multus-certs\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104102 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-kubelet-config\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104123 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/768fa2c0-ba21-4175-b2bc-413f2080040d-agent-certs\") pod \"konnectivity-agent-ppp5w\" (UID: \"768fa2c0-ba21-4175-b2bc-413f2080040d\") " pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104146 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/768fa2c0-ba21-4175-b2bc-413f2080040d-konnectivity-ca\") pod \"konnectivity-agent-ppp5w\" (UID: \"768fa2c0-ba21-4175-b2bc-413f2080040d\") " pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104183 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-systemd\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.104265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104227 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-host\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104255 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-run-netns\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104272 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-systemd\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104325 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-host\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104353 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-dbus\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104373 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104398 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-k8s-cni-cncf-io\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104416 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-hostroot\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104430 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-daemon-config\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104448 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-slash\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104468 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-systemd\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104484 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104501 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104531 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovnkube-config\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104554 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-sys-fs\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104570 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kspv6\" (UniqueName: \"kubernetes.io/projected/47fcab68-8466-42e1-a167-cf728689f513-kube-api-access-kspv6\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.105083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104603 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5knjr\" (UniqueName: \"kubernetes.io/projected/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-kube-api-access-5knjr\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104629 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6j6\" (UniqueName: \"kubernetes.io/projected/3da1224a-2d52-408c-99a2-0662186b0cab-kube-api-access-9w6j6\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104649 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-cni-multus\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104666 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-var-lib-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104691 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104719 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nllz\" (UniqueName: \"kubernetes.io/projected/c078bb29-1f4d-45f5-beed-4d91e9a4a288-kube-api-access-8nllz\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104758 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-kubernetes\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104782 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-system-cni-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104806 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-cni-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104828 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-socket-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104849 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104888 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-kubernetes\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104897 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5hp\" (UniqueName: \"kubernetes.io/projected/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-kube-api-access-tn5hp\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104940 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.104999 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-run\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105041 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-var-lib-kubelet\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.105868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105074 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-kubelet\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105111 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-systemd-units\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105145 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmw52\" (UniqueName: \"kubernetes.io/projected/430dfc42-dfe5-4568-b1bf-4afada865d71-kube-api-access-tmw52\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105177 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-run\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105218 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysconfig\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105259 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysctl-d\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105267 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-var-lib-kubelet\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105407 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysctl-d\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.106656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.105456 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-sysconfig\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.107409 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.107388 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/430dfc42-dfe5-4568-b1bf-4afada865d71-etc-tuned\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.110907 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.110888 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/430dfc42-dfe5-4568-b1bf-4afada865d71-tmp\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.113468 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.113445 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmw52\" (UniqueName: \"kubernetes.io/projected/430dfc42-dfe5-4568-b1bf-4afada865d71-kube-api-access-tmw52\") pod \"tuned-bl9f7\" (UID: \"430dfc42-dfe5-4568-b1bf-4afada865d71\") " pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.205706 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205676 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-run-netns\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.205861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205714 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-dbus\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.205861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205734 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:12.205861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205751 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-k8s-cni-cncf-io\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.205861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205767 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-hostroot\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.205861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205765 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-run-netns\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.205861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205782 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-daemon-config\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.205861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205822 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-slash\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.205861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205848 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-systemd\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205863 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-k8s-cni-cncf-io\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205886 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-hostroot\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.205908 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205893 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-systemd\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205915 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205946 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205955 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.205905 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-slash\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.205970 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs podName:fecbcf2f-03d7-41ad-9585-3d9193c16bc0 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:12.705952634 +0000 UTC m=+2.140937269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs") pod "network-metrics-daemon-r5pzf" (UID: "fecbcf2f-03d7-41ad-9585-3d9193c16bc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206033 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-dbus\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206096 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovnkube-config\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206116 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206125 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-sys-fs\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206153 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kspv6\" (UniqueName: \"kubernetes.io/projected/47fcab68-8466-42e1-a167-cf728689f513-kube-api-access-kspv6\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206179 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5knjr\" (UniqueName: \"kubernetes.io/projected/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-kube-api-access-5knjr\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206205 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6j6\" (UniqueName: \"kubernetes.io/projected/3da1224a-2d52-408c-99a2-0662186b0cab-kube-api-access-9w6j6\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.206247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206196 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-sys-fs\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206237 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-cni-multus\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206275 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-var-lib-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206309 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206322 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-var-lib-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nllz\" (UniqueName: \"kubernetes.io/projected/c078bb29-1f4d-45f5-beed-4d91e9a4a288-kube-api-access-8nllz\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206277 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-cni-multus\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206363 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-daemon-config\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206368 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-system-cni-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206394 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-cni-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206413 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206425 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-system-cni-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206428 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-socket-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206467 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206474 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-cni-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206492 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5hp\" (UniqueName: \"kubernetes.io/projected/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-kube-api-access-tn5hp\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206517 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-socket-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.207088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206517 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206560 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-kubelet\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206602 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206622 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-systemd-units\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206648 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-kubelet\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206648 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206689 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovnkube-script-lib\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206713 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-node-log\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206715 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovnkube-config\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206732 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-systemd-units\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206737 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5e9aa45-eaf8-4317-b370-780af8488466-host-slash\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206774 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-cni-bin\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206780 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-node-log\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206782 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5e9aa45-eaf8-4317-b370-780af8488466-host-slash\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206800 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-conf-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206826 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-device-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206829 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-cni-bin\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.207936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206853 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-cni-binary-copy\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206888 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-conf-dir\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206904 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5e9aa45-eaf8-4317-b370-780af8488466-iptables-alerter-script\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206921 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-device-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206930 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-cnibin\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.206953 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-etc-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207020 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-cni-bin\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207045 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c078bb29-1f4d-45f5-beed-4d91e9a4a288-hosts-file\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207067 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c078bb29-1f4d-45f5-beed-4d91e9a4a288-tmp-dir\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207111 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207114 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-socket-dir-parent\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207157 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207170 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-netns\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207195 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-multus-socket-dir-parent\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207218 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-cni-bin\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207230 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovnkube-script-lib\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.208653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207238 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c078bb29-1f4d-45f5-beed-4d91e9a4a288-hosts-file\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207257 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-netns\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207287 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-cnibin\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207328 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-etc-openvswitch\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207352 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-kubelet\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207419 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-etc-kubernetes\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207448 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207454 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-var-lib-kubelet\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207463 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3da1224a-2d52-408c-99a2-0662186b0cab-cni-binary-copy\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207477 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-env-overrides\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207498 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-etc-kubernetes\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207505 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-cni-netd\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207508 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c078bb29-1f4d-45f5-beed-4d91e9a4a288-tmp-dir\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207522 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207531 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207557 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207563 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-host-cni-netd\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207583 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-cnibin\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.209228 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.207619 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207620 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207637 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-os-release\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.207661 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret podName:ec9d1d25-83d7-4926-abe8-191d4e3fefe7 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:12.707648176 +0000 UTC m=+2.142632780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret") pod "global-pull-secret-syncer-wrmgv" (UID: "ec9d1d25-83d7-4926-abe8-191d4e3fefe7") : object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207667 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-cnibin\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207687 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-os-release\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207705 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25f418cb-ca33-4715-8acf-7a84b44371cc-cni-binary-copy\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207708 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-os-release\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207720 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-host\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207737 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-os-release\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207745 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-host\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207747 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-serviceca\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207686 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5e9aa45-eaf8-4317-b370-780af8488466-iptables-alerter-script\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207776 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-system-cni-dir\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207794 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdh9\" (UniqueName: \"kubernetes.io/projected/f5e9aa45-eaf8-4317-b370-780af8488466-kube-api-access-8hdh9\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207812 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klbg\" (UniqueName: \"kubernetes.io/projected/25f418cb-ca33-4715-8acf-7a84b44371cc-kube-api-access-5klbg\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207829 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-ovn\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.209768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207850 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-log-socket\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207892 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-env-overrides\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207897 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-log-socket\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovn-node-metrics-cert\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207940 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csngt\" (UniqueName: \"kubernetes.io/projected/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-kube-api-access-csngt\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.207966 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-registration-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208005 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-multus-certs\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208031 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-kubelet-config\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208060 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47fcab68-8466-42e1-a167-cf728689f513-registration-dir\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208060 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/768fa2c0-ba21-4175-b2bc-413f2080040d-agent-certs\") pod \"konnectivity-agent-ppp5w\" (UID: \"768fa2c0-ba21-4175-b2bc-413f2080040d\") " pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208113 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/768fa2c0-ba21-4175-b2bc-413f2080040d-konnectivity-ca\") pod \"konnectivity-agent-ppp5w\" (UID: \"768fa2c0-ba21-4175-b2bc-413f2080040d\") " pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208118 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/25f418cb-ca33-4715-8acf-7a84b44371cc-host-run-multus-certs\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208003 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3da1224a-2d52-408c-99a2-0662186b0cab-system-cni-dir\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208234 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-run-ovn\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208279 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-kubelet-config\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208329 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-serviceca\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208334 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25f418cb-ca33-4715-8acf-7a84b44371cc-cni-binary-copy\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.210366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.208656 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/768fa2c0-ba21-4175-b2bc-413f2080040d-konnectivity-ca\") pod \"konnectivity-agent-ppp5w\" (UID: \"768fa2c0-ba21-4175-b2bc-413f2080040d\") " pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:12.211191 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.210378 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-ovn-node-metrics-cert\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.211191 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.211107 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/768fa2c0-ba21-4175-b2bc-413f2080040d-agent-certs\") pod \"konnectivity-agent-ppp5w\" (UID: \"768fa2c0-ba21-4175-b2bc-413f2080040d\") " pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:12.218450 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.218426 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 14:30:12.218450 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.218449 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 14:30:12.218629 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.218459 2550 projected.go:196] Error preparing data for projected volume kube-api-access-z79xl for pod openshift-network-diagnostics/network-check-target-m9xxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:12.218629 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.218516 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl podName:9ce2c26e-1d31-4e56-a620-1a0cfb60da47 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:12.718502395 +0000 UTC m=+2.153487001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z79xl" (UniqueName: "kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl") pod "network-check-target-m9xxq" (UID: "9ce2c26e-1d31-4e56-a620-1a0cfb60da47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:12.220722 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.220691 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nllz\" (UniqueName: \"kubernetes.io/projected/c078bb29-1f4d-45f5-beed-4d91e9a4a288-kube-api-access-8nllz\") pod \"node-resolver-w4pnw\" (UID: \"c078bb29-1f4d-45f5-beed-4d91e9a4a288\") " pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.221480 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.221453 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdh9\" (UniqueName: \"kubernetes.io/projected/f5e9aa45-eaf8-4317-b370-780af8488466-kube-api-access-8hdh9\") pod \"iptables-alerter-dgqsn\" (UID: \"f5e9aa45-eaf8-4317-b370-780af8488466\") " pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.222022 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.221974 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5knjr\" (UniqueName: \"kubernetes.io/projected/6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a-kube-api-access-5knjr\") pod \"node-ca-mrt4j\" (UID: \"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a\") " pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.222443 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.222412 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klbg\" (UniqueName: \"kubernetes.io/projected/25f418cb-ca33-4715-8acf-7a84b44371cc-kube-api-access-5klbg\") pod \"multus-j5vxr\" (UID: \"25f418cb-ca33-4715-8acf-7a84b44371cc\") " pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.222627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.222604 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5hp\" (UniqueName: \"kubernetes.io/projected/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-kube-api-access-tn5hp\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:12.222689 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.222613 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6j6\" (UniqueName: \"kubernetes.io/projected/3da1224a-2d52-408c-99a2-0662186b0cab-kube-api-access-9w6j6\") pod \"multus-additional-cni-plugins-dkvx6\" (UID: \"3da1224a-2d52-408c-99a2-0662186b0cab\") " pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.222689 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.222645 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csngt\" (UniqueName: \"kubernetes.io/projected/09b8e867-76c5-4c35-b287-b2a2a1ee17a5-kube-api-access-csngt\") pod \"ovnkube-node-6998d\" (UID: \"09b8e867-76c5-4c35-b287-b2a2a1ee17a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.223116 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.223097 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kspv6\" (UniqueName: \"kubernetes.io/projected/47fcab68-8466-42e1-a167-cf728689f513-kube-api-access-kspv6\") pod \"aws-ebs-csi-driver-node-zlwsq\" (UID: \"47fcab68-8466-42e1-a167-cf728689f513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.238272 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.238236 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acd846e89bc70267fc722d6edc8fc43.slice/crio-7e46d2a21e62445a67258b1532e6424d6392056d26676557bae01a885ccf643f WatchSource:0}: Error finding container 7e46d2a21e62445a67258b1532e6424d6392056d26676557bae01a885ccf643f: Status 404 returned error can't find the container with id 7e46d2a21e62445a67258b1532e6424d6392056d26676557bae01a885ccf643f Jul 14 14:30:12.238713 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.238693 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb675f1bf43f9d3e7b9ee8182f17d78ea.slice/crio-002b9658fd5b0f85fddabc6e39c4ac7354e9930a00b99999209e6f686f42387d WatchSource:0}: Error finding container 002b9658fd5b0f85fddabc6e39c4ac7354e9930a00b99999209e6f686f42387d: Status 404 returned error can't find the container with id 002b9658fd5b0f85fddabc6e39c4ac7354e9930a00b99999209e6f686f42387d Jul 14 14:30:12.244300 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.244284 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 14:30:12.356982 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.356928 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" Jul 14 14:30:12.364289 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.362180 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 14:30:12.367950 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.367927 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430dfc42_dfe5_4568_b1bf_4afada865d71.slice/crio-26b66a6fea289169d9a0c573169b21fdf4e406517672d2575eca510057824a19 WatchSource:0}: Error finding container 26b66a6fea289169d9a0c573169b21fdf4e406517672d2575eca510057824a19: Status 404 returned error can't find the container with id 26b66a6fea289169d9a0c573169b21fdf4e406517672d2575eca510057824a19 Jul 14 14:30:12.373055 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.373026 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5vxr" Jul 14 14:30:12.381392 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.381368 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:12.381928 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.381908 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f418cb_ca33_4715_8acf_7a84b44371cc.slice/crio-5d10a4aa396b2c0749363dd97dada56a6b183d7c511791d9486e8c4c1f970dcc WatchSource:0}: Error finding container 5d10a4aa396b2c0749363dd97dada56a6b183d7c511791d9486e8c4c1f970dcc: Status 404 returned error can't find the container with id 5d10a4aa396b2c0749363dd97dada56a6b183d7c511791d9486e8c4c1f970dcc Jul 14 14:30:12.389899 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.389872 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b8e867_76c5_4c35_b287_b2a2a1ee17a5.slice/crio-b80f2774134d02bd6831170523b2099e5e78782079f84dda9aaa5189ea72f657 WatchSource:0}: Error finding container b80f2774134d02bd6831170523b2099e5e78782079f84dda9aaa5189ea72f657: Status 404 returned error can't find the container with id b80f2774134d02bd6831170523b2099e5e78782079f84dda9aaa5189ea72f657 Jul 14 14:30:12.408381 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.408318 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:12.409858 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.409838 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 14:30:12.417030 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.417002 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768fa2c0_ba21_4175_b2bc_413f2080040d.slice/crio-ce45593217a320ae3ab8206ce0970fa0c64f0f9c809fdb25058022ec4c139641 WatchSource:0}: Error finding container ce45593217a320ae3ab8206ce0970fa0c64f0f9c809fdb25058022ec4c139641: Status 404 returned error can't find the container with id ce45593217a320ae3ab8206ce0970fa0c64f0f9c809fdb25058022ec4c139641 Jul 14 14:30:12.422225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.422204 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" Jul 14 14:30:12.431846 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.431822 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fcab68_8466_42e1_a167_cf728689f513.slice/crio-c8acc54bfb8f8132682b85a66b2eed8396bbf69ae5223c2b207cabeeeb9b2279 WatchSource:0}: Error finding container c8acc54bfb8f8132682b85a66b2eed8396bbf69ae5223c2b207cabeeeb9b2279: Status 404 returned error can't find the container with id c8acc54bfb8f8132682b85a66b2eed8396bbf69ae5223c2b207cabeeeb9b2279 Jul 14 14:30:12.443305 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.443277 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mrt4j" Jul 14 14:30:12.451780 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.451754 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2c2d3f_0746_4db3_9fd8_f486ecdaf82a.slice/crio-8276a4cd346c756e34cfb0bb9b38c7ecef4a3208f08e8b5a9b97b50b98a312dd WatchSource:0}: Error finding container 8276a4cd346c756e34cfb0bb9b38c7ecef4a3208f08e8b5a9b97b50b98a312dd: Status 404 returned error can't find the container with id 8276a4cd346c756e34cfb0bb9b38c7ecef4a3208f08e8b5a9b97b50b98a312dd Jul 14 14:30:12.454012 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.453995 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" Jul 14 14:30:12.462679 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.462653 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da1224a_2d52_408c_99a2_0662186b0cab.slice/crio-352ffebfa9c77b9e599c4367ae5899d9e26784e812bc0e3083476ab19336321c WatchSource:0}: Error finding container 352ffebfa9c77b9e599c4367ae5899d9e26784e812bc0e3083476ab19336321c: Status 404 returned error can't find the container with id 352ffebfa9c77b9e599c4367ae5899d9e26784e812bc0e3083476ab19336321c Jul 14 14:30:12.470438 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.470419 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w4pnw" Jul 14 14:30:12.476864 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.476845 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dgqsn" Jul 14 14:30:12.479676 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.479651 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc078bb29_1f4d_45f5_beed_4d91e9a4a288.slice/crio-861a97807206f05714570beba2c54753c9c52899df3a9728a6225f694948f20a WatchSource:0}: Error finding container 861a97807206f05714570beba2c54753c9c52899df3a9728a6225f694948f20a: Status 404 returned error can't find the container with id 861a97807206f05714570beba2c54753c9c52899df3a9728a6225f694948f20a Jul 14 14:30:12.485752 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:12.485728 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e9aa45_eaf8_4317_b370_780af8488466.slice/crio-14cf13e4621072e80df811a3f1b30a52e7a2681f568e6ff7aa40897d61b4b5dd WatchSource:0}: Error finding container 14cf13e4621072e80df811a3f1b30a52e7a2681f568e6ff7aa40897d61b4b5dd: Status 404 returned error can't find the container with id 14cf13e4621072e80df811a3f1b30a52e7a2681f568e6ff7aa40897d61b4b5dd Jul 14 14:30:12.712476 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.712395 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:12.712476 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.712437 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:12.712649 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.712554 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:12.712649 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.712608 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret podName:ec9d1d25-83d7-4926-abe8-191d4e3fefe7 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:13.712594717 +0000 UTC m=+3.147579319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret") pod "global-pull-secret-syncer-wrmgv" (UID: "ec9d1d25-83d7-4926-abe8-191d4e3fefe7") : object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:12.712649 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.712554 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:12.712750 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.712688 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs podName:fecbcf2f-03d7-41ad-9585-3d9193c16bc0 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:13.71267502 +0000 UTC m=+3.147659638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs") pod "network-metrics-daemon-r5pzf" (UID: "fecbcf2f-03d7-41ad-9585-3d9193c16bc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:12.812756 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:12.812721 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:12.812935 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.812898 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 14:30:12.812935 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.812921 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 14:30:12.812935 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.812933 2550 projected.go:196] Error preparing data for projected volume kube-api-access-z79xl for pod openshift-network-diagnostics/network-check-target-m9xxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:12.813122 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:12.813025 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl podName:9ce2c26e-1d31-4e56-a620-1a0cfb60da47 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:13.813005495 +0000 UTC m=+3.247990119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z79xl" (UniqueName: "kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl") pod "network-check-target-m9xxq" (UID: "9ce2c26e-1d31-4e56-a620-1a0cfb60da47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:13.098149 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.097939 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 14:25:12 +0000 UTC" deadline="2027-12-08 06:58:31.519628139 +0000 UTC" Jul 14 14:30:13.098149 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.098019 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12280h28m18.421613954s" Jul 14 14:30:13.121116 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.120998 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dgqsn" event={"ID":"f5e9aa45-eaf8-4317-b370-780af8488466","Type":"ContainerStarted","Data":"14cf13e4621072e80df811a3f1b30a52e7a2681f568e6ff7aa40897d61b4b5dd"} Jul 14 14:30:13.122663 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.122636 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w4pnw" event={"ID":"c078bb29-1f4d-45f5-beed-4d91e9a4a288","Type":"ContainerStarted","Data":"861a97807206f05714570beba2c54753c9c52899df3a9728a6225f694948f20a"} Jul 14 14:30:13.127397 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.127321 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" event={"ID":"3da1224a-2d52-408c-99a2-0662186b0cab","Type":"ContainerStarted","Data":"352ffebfa9c77b9e599c4367ae5899d9e26784e812bc0e3083476ab19336321c"} Jul 14 14:30:13.132805 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.132770 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mrt4j" event={"ID":"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a","Type":"ContainerStarted","Data":"8276a4cd346c756e34cfb0bb9b38c7ecef4a3208f08e8b5a9b97b50b98a312dd"} Jul 14 14:30:13.135002 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.134953 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" event={"ID":"47fcab68-8466-42e1-a167-cf728689f513","Type":"ContainerStarted","Data":"c8acc54bfb8f8132682b85a66b2eed8396bbf69ae5223c2b207cabeeeb9b2279"} Jul 14 14:30:13.137218 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.137192 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"b80f2774134d02bd6831170523b2099e5e78782079f84dda9aaa5189ea72f657"} Jul 14 14:30:13.140086 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.140060 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5vxr" event={"ID":"25f418cb-ca33-4715-8acf-7a84b44371cc","Type":"ContainerStarted","Data":"5d10a4aa396b2c0749363dd97dada56a6b183d7c511791d9486e8c4c1f970dcc"} Jul 14 14:30:13.142296 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.142236 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ppp5w" event={"ID":"768fa2c0-ba21-4175-b2bc-413f2080040d","Type":"ContainerStarted","Data":"ce45593217a320ae3ab8206ce0970fa0c64f0f9c809fdb25058022ec4c139641"} Jul 14 14:30:13.143979 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.143948 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" event={"ID":"430dfc42-dfe5-4568-b1bf-4afada865d71","Type":"ContainerStarted","Data":"26b66a6fea289169d9a0c573169b21fdf4e406517672d2575eca510057824a19"} Jul 14 14:30:13.148947 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.148927 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" event={"ID":"b675f1bf43f9d3e7b9ee8182f17d78ea","Type":"ContainerStarted","Data":"002b9658fd5b0f85fddabc6e39c4ac7354e9930a00b99999209e6f686f42387d"} Jul 14 14:30:13.151571 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.151548 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" event={"ID":"1acd846e89bc70267fc722d6edc8fc43","Type":"ContainerStarted","Data":"7e46d2a21e62445a67258b1532e6424d6392056d26676557bae01a885ccf643f"} Jul 14 14:30:13.205860 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.205826 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 14:30:13.721705 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.721662 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:13.721915 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.721736 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:13.721984 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:13.721913 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:13.721984 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:13.721979 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs podName:fecbcf2f-03d7-41ad-9585-3d9193c16bc0 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:15.721960856 +0000 UTC m=+5.156945481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs") pod "network-metrics-daemon-r5pzf" (UID: "fecbcf2f-03d7-41ad-9585-3d9193c16bc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:13.722428 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:13.722389 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:13.722503 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:13.722444 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret podName:ec9d1d25-83d7-4926-abe8-191d4e3fefe7 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:15.722428043 +0000 UTC m=+5.157412660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret") pod "global-pull-secret-syncer-wrmgv" (UID: "ec9d1d25-83d7-4926-abe8-191d4e3fefe7") : object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:13.822850 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:13.822787 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:13.823070 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:13.823051 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 14:30:13.823135 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:13.823082 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 14:30:13.823135 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:13.823097 2550 projected.go:196] Error preparing data for projected volume kube-api-access-z79xl for pod openshift-network-diagnostics/network-check-target-m9xxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:13.823206 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:13.823159 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl podName:9ce2c26e-1d31-4e56-a620-1a0cfb60da47 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:15.823139593 +0000 UTC m=+5.258124197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z79xl" (UniqueName: "kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl") pod "network-check-target-m9xxq" (UID: "9ce2c26e-1d31-4e56-a620-1a0cfb60da47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:14.098569 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:14.098462 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 14:25:12 +0000 UTC" deadline="2028-02-22 09:23:48.65243762 +0000 UTC" Jul 14 14:30:14.098569 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:14.098520 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14106h53m34.553922689s" Jul 14 14:30:14.113971 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:14.113941 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:14.114200 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:14.114126 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:14.114679 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:14.114658 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:14.114831 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:14.114784 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:14.114905 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:14.114864 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:14.114967 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:14.114951 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:14.527174 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:14.527092 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 14:30:15.743428 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:15.742710 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:15.743428 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:15.742779 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:15.743428 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:15.742927 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:15.743428 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:15.742934 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:15.743428 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:15.742992 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs podName:fecbcf2f-03d7-41ad-9585-3d9193c16bc0 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:19.742974015 +0000 UTC m=+9.177958633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs") pod "network-metrics-daemon-r5pzf" (UID: "fecbcf2f-03d7-41ad-9585-3d9193c16bc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:15.743428 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:15.743020 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret podName:ec9d1d25-83d7-4926-abe8-191d4e3fefe7 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:19.742999357 +0000 UTC m=+9.177983964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret") pod "global-pull-secret-syncer-wrmgv" (UID: "ec9d1d25-83d7-4926-abe8-191d4e3fefe7") : object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:15.844182 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:15.843438 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:15.844182 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:15.843639 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 14:30:15.844182 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:15.843658 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 14:30:15.844182 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:15.843671 2550 projected.go:196] Error preparing data for projected volume kube-api-access-z79xl for pod openshift-network-diagnostics/network-check-target-m9xxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:15.844182 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:15.843732 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl podName:9ce2c26e-1d31-4e56-a620-1a0cfb60da47 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:19.8437139 +0000 UTC m=+9.278698506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z79xl" (UniqueName: "kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl") pod "network-check-target-m9xxq" (UID: "9ce2c26e-1d31-4e56-a620-1a0cfb60da47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:16.114092 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:16.114060 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:16.114274 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:16.114205 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:16.114639 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:16.114618 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:16.114732 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:16.114715 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:16.114810 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:16.114783 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:16.114874 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:16.114859 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:18.114590 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:18.114552 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:18.115180 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:18.114695 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:18.115180 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:18.114771 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:18.115180 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:18.114861 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:18.115180 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:18.114907 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:18.115180 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:18.114980 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:19.781818 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:19.781459 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:19.781818 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:19.781526 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:19.781818 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:19.781643 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:19.781818 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:19.781661 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:19.781818 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:19.781719 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs podName:fecbcf2f-03d7-41ad-9585-3d9193c16bc0 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:27.781695452 +0000 UTC m=+17.216680116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs") pod "network-metrics-daemon-r5pzf" (UID: "fecbcf2f-03d7-41ad-9585-3d9193c16bc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:19.781818 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:19.781738 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret podName:ec9d1d25-83d7-4926-abe8-191d4e3fefe7 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:27.781728435 +0000 UTC m=+17.216713045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret") pod "global-pull-secret-syncer-wrmgv" (UID: "ec9d1d25-83d7-4926-abe8-191d4e3fefe7") : object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:19.882883 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:19.882842 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:19.883180 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:19.883119 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 14:30:19.883180 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:19.883156 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 14:30:19.883180 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:19.883172 2550 projected.go:196] Error preparing data for projected volume kube-api-access-z79xl for pod openshift-network-diagnostics/network-check-target-m9xxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:19.883384 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:19.883266 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl podName:9ce2c26e-1d31-4e56-a620-1a0cfb60da47 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:27.883243025 +0000 UTC m=+17.318227652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z79xl" (UniqueName: "kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl") pod "network-check-target-m9xxq" (UID: "9ce2c26e-1d31-4e56-a620-1a0cfb60da47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:20.114838 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:20.114298 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:20.114838 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:20.114448 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:20.115100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:20.114842 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:20.115100 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:20.114974 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:20.115100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:20.115013 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:20.115265 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:20.115199 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:22.115019 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:22.114478 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:22.115019 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:22.114618 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:22.115019 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:22.115014 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:22.115597 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:22.115044 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:22.115597 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:22.115130 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:22.115597 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:22.115227 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:24.113979 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:24.113933 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:24.114420 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:24.114052 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:24.114420 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:24.114088 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:24.114420 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:24.114212 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:24.114420 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:24.114263 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:24.114420 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:24.114338 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:26.114743 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:26.114702 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:26.114743 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:26.114731 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:26.115216 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:26.114735 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:26.115216 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:26.114832 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:26.115216 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:26.114917 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:26.115216 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:26.114984 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:27.843757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:27.843717 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:27.844257 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:27.843771 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:27.844257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:27.843879 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:27.844257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:27.843894 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:27.844257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:27.843940 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs podName:fecbcf2f-03d7-41ad-9585-3d9193c16bc0 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:43.843922641 +0000 UTC m=+33.278907245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs") pod "network-metrics-daemon-r5pzf" (UID: "fecbcf2f-03d7-41ad-9585-3d9193c16bc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:27.844257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:27.843955 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret podName:ec9d1d25-83d7-4926-abe8-191d4e3fefe7 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:43.843948901 +0000 UTC m=+33.278933502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret") pod "global-pull-secret-syncer-wrmgv" (UID: "ec9d1d25-83d7-4926-abe8-191d4e3fefe7") : object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:27.944268 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:27.944233 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:27.944452 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:27.944420 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 14:30:27.944452 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:27.944445 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 14:30:27.944552 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:27.944458 2550 projected.go:196] Error preparing data for projected volume kube-api-access-z79xl for pod openshift-network-diagnostics/network-check-target-m9xxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:27.944552 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:27.944522 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl podName:9ce2c26e-1d31-4e56-a620-1a0cfb60da47 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:43.944503278 +0000 UTC m=+33.379487892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z79xl" (UniqueName: "kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl") pod "network-check-target-m9xxq" (UID: "9ce2c26e-1d31-4e56-a620-1a0cfb60da47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:28.113864 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:28.113767 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:28.114064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:28.113775 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:28.114064 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:28.113921 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:28.114064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:28.113775 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:28.114064 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:28.113990 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:28.114274 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:28.114088 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:30.114183 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:30.114143 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:30.114636 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:30.114143 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:30.114636 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:30.114291 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:30.114636 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:30.114143 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:30.114636 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:30.114358 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:30.114636 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:30.114431 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:32.115018 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:32.114773 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:32.115454 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:32.114814 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:32.115454 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:32.115122 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:32.115454 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:32.114820 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:32.115454 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:32.115208 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:32.115454 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:32.115269 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:33.218661 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:33.218411 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" event={"ID":"1acd846e89bc70267fc722d6edc8fc43","Type":"ContainerStarted","Data":"351cc3253c295225bac952a452e5069433c8d5affae9c8058cfd3226f1af0ee7"} Jul 14 14:30:33.220761 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:33.220719 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"5e7ee2b7dce6e11fe82549c45f2ed1e1db047c5db1d4e4359497532ffc2c7521"} Jul 14 14:30:33.223058 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:33.223025 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5vxr" event={"ID":"25f418cb-ca33-4715-8acf-7a84b44371cc","Type":"ContainerStarted","Data":"5040e68d1de9a2992fcbdb4959ca16c46f511c5cff7ad7fefa04b3273c5b6df9"} Jul 14 14:30:33.225465 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:33.225435 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" event={"ID":"430dfc42-dfe5-4568-b1bf-4afada865d71","Type":"ContainerStarted","Data":"287f2c887fd0d4d3e112847b8bd317ccabce57ca7b22cb4b9d8189507a3a0bdb"} Jul 14 14:30:33.234865 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:33.234808 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.ec2.internal" podStartSLOduration=21.234788251 podStartE2EDuration="21.234788251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:33.233902805 +0000 UTC m=+22.668887432" watchObservedRunningTime="2026-07-14 14:30:33.234788251 +0000 UTC m=+22.669772875" Jul 14 14:30:33.276048 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:33.275067 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bl9f7" podStartSLOduration=1.64668645 podStartE2EDuration="22.275044103s" totalImagesPullingTime="20.628357653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:33.251047806 +0000 UTC m=+22.686032430" watchObservedRunningTime="2026-07-14 14:30:33.275044103 +0000 UTC m=+22.710028928" Jul 14 14:30:34.114624 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.114425 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:34.114901 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.114473 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:34.114901 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:34.114780 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:34.114901 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.114473 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:34.114901 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:34.114682 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:34.114901 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:34.114894 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:34.229627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.229597 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w4pnw" event={"ID":"c078bb29-1f4d-45f5-beed-4d91e9a4a288","Type":"ContainerStarted","Data":"6300a0c7b91c50c60b68689347694c4364965e85f12bbb23a95dfac91eb85d8e"} Jul 14 14:30:34.231258 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.231235 2550 generic.go:356] "Generic (PLEG): container finished" podID="3da1224a-2d52-408c-99a2-0662186b0cab" containerID="4410c4e6d1ccb17cb0ebf7dc1649b39ae8b58613f5a973d8cf03f994eae36f9d" exitCode=0 Jul 14 14:30:34.231385 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.231271 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" event={"ID":"3da1224a-2d52-408c-99a2-0662186b0cab","Type":"ContainerDied","Data":"4410c4e6d1ccb17cb0ebf7dc1649b39ae8b58613f5a973d8cf03f994eae36f9d"} Jul 14 14:30:34.232595 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.232575 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mrt4j" event={"ID":"6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a","Type":"ContainerStarted","Data":"bd3ab55df8920102529b88a5e35ff85b9a852b251c8400b35e1336494a2b472f"} Jul 14 14:30:34.234082 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.234037 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" event={"ID":"47fcab68-8466-42e1-a167-cf728689f513","Type":"ContainerStarted","Data":"f7697d69871d53a72a3258f021c5ff67665004bd0fbb367ab466df72c8e7c84c"} Jul 14 14:30:34.236503 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.236487 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:30:34.236798 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.236781 2550 generic.go:356] "Generic (PLEG): container finished" podID="09b8e867-76c5-4c35-b287-b2a2a1ee17a5" containerID="2539a20b52152e4b18e5ee202f667f85018837298dd1ce095ab0a8fe58875f69" exitCode=1 Jul 14 14:30:34.236857 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.236840 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"37f3aee6d84fe61fd2fcd9009fee5b2f7d55f7f2ccc1d62d861324690f32fdf3"} Jul 14 14:30:34.236938 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.236903 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"ef64a1d0d32814a7b2e8a69df82b67f4564b91d98a3fe39868dd15247a9a6cb1"} Jul 14 14:30:34.237046 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.237031 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"9aa8a2101bebb567dbd4af478c492a19841a65f82b1b5a492db5290e7a11a75e"} Jul 14 14:30:34.237092 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.237081 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"bef71baa6d4c8ba1cec1064271d2d02e2c98ae35293945bafe6ff2b3039d4d57"} Jul 14 14:30:34.237125 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.237116 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerDied","Data":"2539a20b52152e4b18e5ee202f667f85018837298dd1ce095ab0a8fe58875f69"} Jul 14 14:30:34.238150 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.238131 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ppp5w" event={"ID":"768fa2c0-ba21-4175-b2bc-413f2080040d","Type":"ContainerStarted","Data":"f7808b9809d276f52266eea1beae25526049b43303ad5bb6e04f14d0a33347e4"} Jul 14 14:30:34.239425 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.239407 2550 generic.go:356] "Generic (PLEG): container finished" podID="b675f1bf43f9d3e7b9ee8182f17d78ea" containerID="50acc5e167ea5bb68bdf754c76f4dbd864cbf6413a7ace063a544e0a820899a9" exitCode=0 Jul 14 14:30:34.239522 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.239503 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" event={"ID":"b675f1bf43f9d3e7b9ee8182f17d78ea","Type":"ContainerDied","Data":"50acc5e167ea5bb68bdf754c76f4dbd864cbf6413a7ace063a544e0a820899a9"} Jul 14 14:30:34.245533 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.245494 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w4pnw" podStartSLOduration=1.746483247 podStartE2EDuration="22.245483184s" totalImagesPullingTime="20.498999937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:34.245243146 +0000 UTC m=+23.680227769" watchObservedRunningTime="2026-07-14 14:30:34.245483184 +0000 UTC m=+23.680467807" Jul 14 14:30:34.245627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.245556 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j5vxr" podStartSLOduration=2.593496386 podStartE2EDuration="23.245551931s" totalImagesPullingTime="20.652055545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:33.274673902 +0000 UTC m=+22.709658524" watchObservedRunningTime="2026-07-14 14:30:34.245551931 +0000 UTC m=+23.680536555" Jul 14 14:30:34.259909 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.259873 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ppp5w" podStartSLOduration=1.694544736 podStartE2EDuration="22.259861163s" totalImagesPullingTime="20.565316427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:34.259614773 +0000 UTC m=+23.694599394" watchObservedRunningTime="2026-07-14 14:30:34.259861163 +0000 UTC m=+23.694845811" Jul 14 14:30:34.277192 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.277152 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mrt4j" podStartSLOduration=1.7436921010000002 podStartE2EDuration="22.277139277s" totalImagesPullingTime="20.533447176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:34.27680291 +0000 UTC m=+23.711787534" watchObservedRunningTime="2026-07-14 14:30:34.277139277 +0000 UTC m=+23.712123928" Jul 14 14:30:34.366550 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:34.366525 2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 14:30:35.071033 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:35.070904 2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T14:30:34.366543656Z","UUID":"b665a45e-83e2-4932-b305-65ccc5e38481","Handler":null,"Name":"","Endpoint":""} Jul 14 14:30:35.072511 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:35.072488 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 14:30:35.072620 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:35.072533 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 14:30:35.244190 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:35.244140 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" event={"ID":"47fcab68-8466-42e1-a167-cf728689f513","Type":"ContainerStarted","Data":"eb3daec2cf4bd21bc648d8ac5060d9f0fa88cc6def36471a61a310f7dc7db908"} Jul 14 14:30:35.246272 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:35.246239 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" event={"ID":"b675f1bf43f9d3e7b9ee8182f17d78ea","Type":"ContainerStarted","Data":"9a0e501706e53a7f9cb4e716f0f264ba1a83f4a742c838481d1c57167b8269ab"} Jul 14 14:30:35.247709 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:35.247677 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dgqsn" event={"ID":"f5e9aa45-eaf8-4317-b370-780af8488466","Type":"ContainerStarted","Data":"d0b552b223679eb75e738231e2e55177bc4e0f3e05b5ca1c21f3c7963dc261a3"} Jul 14 14:30:35.262389 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:35.262333 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.ec2.internal" podStartSLOduration=23.262320634 podStartE2EDuration="23.262320634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:35.261905502 +0000 UTC m=+24.696890125" watchObservedRunningTime="2026-07-14 14:30:35.262320634 +0000 UTC m=+24.697305258" Jul 14 14:30:35.276192 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:35.276146 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dgqsn" podStartSLOduration=2.79387983 podStartE2EDuration="23.276130007s" totalImagesPullingTime="20.482250177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:35.275798493 +0000 UTC m=+24.710783117" watchObservedRunningTime="2026-07-14 14:30:35.276130007 +0000 UTC m=+24.711114631" Jul 14 14:30:36.114025 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:36.113991 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:36.114025 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:36.114020 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:36.114265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:36.114002 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:36.114265 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:36.114146 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:36.114265 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:36.114237 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:36.114381 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:36.114306 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:36.254190 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:36.254156 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:30:36.254634 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:36.254578 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"b6a0df81d9a9221fbe31a0ca24c8c9549235ce0f0b7aaa94cdc5c6d77c3bcb36"} Jul 14 14:30:37.259686 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:37.259439 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" event={"ID":"47fcab68-8466-42e1-a167-cf728689f513","Type":"ContainerStarted","Data":"b23e6a699eb55b7c3edcf06c1dc872605824e4bd6209ae049d146bc397b0f763"} Jul 14 14:30:37.278325 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:37.278268 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlwsq" podStartSLOduration=1.844028068 podStartE2EDuration="25.278251405s" totalImagesPullingTime="23.434223337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:37.277475999 +0000 UTC m=+26.712460624" watchObservedRunningTime="2026-07-14 14:30:37.278251405 +0000 UTC m=+26.713236028" Jul 14 14:30:38.113845 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:38.113810 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:38.114040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:38.113917 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:38.114040 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:38.113923 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:38.114040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:38.113955 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:38.114204 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:38.114079 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:38.114204 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:38.114150 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:38.373588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:38.373502 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:38.374292 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:38.374263 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:39.265801 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:39.265595 2550 generic.go:356] "Generic (PLEG): container finished" podID="3da1224a-2d52-408c-99a2-0662186b0cab" containerID="4f8835bdc4ef371328eb145df85c28d28410b372f3a7b77789e59172446e2726" exitCode=0 Jul 14 14:30:39.266015 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:39.265680 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" event={"ID":"3da1224a-2d52-408c-99a2-0662186b0cab","Type":"ContainerDied","Data":"4f8835bdc4ef371328eb145df85c28d28410b372f3a7b77789e59172446e2726"} Jul 14 14:30:39.269300 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:39.269280 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:30:39.269691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:39.269661 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"b0d106cfcb6d5d3c4675821d612d3c9214a62aa1a941d930a240cb03ef62be2e"} Jul 14 14:30:39.270030 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:39.270002 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:39.270216 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:39.270202 2550 scope.go:117] "RemoveContainer" containerID="2539a20b52152e4b18e5ee202f667f85018837298dd1ce095ab0a8fe58875f69" Jul 14 14:30:39.270433 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:39.270411 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ppp5w" Jul 14 14:30:40.114165 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.114135 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:40.114983 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.114248 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:40.114983 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:40.114262 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:40.114983 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.114276 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:40.114983 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:40.114343 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:40.114983 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:40.114410 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:40.275646 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.275621 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:30:40.276100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.276067 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" event={"ID":"09b8e867-76c5-4c35-b287-b2a2a1ee17a5","Type":"ContainerStarted","Data":"d9575d5466b10194dabfcb0f508e7757e37eb1a20b64ba3c0661460c6e1ecbef"} Jul 14 14:30:40.276228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.276161 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:40.276297 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.276234 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:40.276503 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.276425 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:40.293955 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.293921 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:40.294153 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.294134 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:30:40.304035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.303990 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" podStartSLOduration=8.650030974 podStartE2EDuration="29.303955857s" totalImagesPullingTime="20.653924883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:40.302164088 +0000 UTC m=+29.737148712" watchObservedRunningTime="2026-07-14 14:30:40.303955857 +0000 UTC m=+29.738940484" Jul 14 14:30:40.500015 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.499991 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wrmgv"] Jul 14 14:30:40.500099 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.500085 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:40.500187 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:40.500165 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:40.503802 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.503775 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m9xxq"] Jul 14 14:30:40.503902 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.503867 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:40.503990 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:40.503955 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:40.504468 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.504443 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5pzf"] Jul 14 14:30:40.504546 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:40.504535 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:40.504630 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:40.504615 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:41.279942 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:41.279904 2550 generic.go:356] "Generic (PLEG): container finished" podID="3da1224a-2d52-408c-99a2-0662186b0cab" containerID="6964a06f7e167fe3c41842a2d2c58d6aa30241e11b40e4447b5c916b5dc2b736" exitCode=0 Jul 14 14:30:41.280463 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:41.279997 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" event={"ID":"3da1224a-2d52-408c-99a2-0662186b0cab","Type":"ContainerDied","Data":"6964a06f7e167fe3c41842a2d2c58d6aa30241e11b40e4447b5c916b5dc2b736"} Jul 14 14:30:42.114410 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:42.114375 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:42.114410 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:42.114409 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:42.114635 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:42.114375 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:42.114635 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:42.114516 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:42.114635 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:42.114558 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:42.114771 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:42.114742 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:43.287023 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:43.286824 2550 generic.go:356] "Generic (PLEG): container finished" podID="3da1224a-2d52-408c-99a2-0662186b0cab" containerID="0bee782f8bdc5a02bf77ee56d8eebf04bb70c8b784691a6436fd5eeecd444040" exitCode=0 Jul 14 14:30:43.287450 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:43.286905 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" event={"ID":"3da1224a-2d52-408c-99a2-0662186b0cab","Type":"ContainerDied","Data":"0bee782f8bdc5a02bf77ee56d8eebf04bb70c8b784691a6436fd5eeecd444040"} Jul 14 14:30:43.869211 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:43.869179 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:43.869347 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:43.869220 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:43.869388 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:43.869347 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:43.869426 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:43.869393 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:43.869426 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:43.869413 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret podName:ec9d1d25-83d7-4926-abe8-191d4e3fefe7 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:15.869398994 +0000 UTC m=+65.304383596 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret") pod "global-pull-secret-syncer-wrmgv" (UID: "ec9d1d25-83d7-4926-abe8-191d4e3fefe7") : object "kube-system"/"original-pull-secret" not registered Jul 14 14:30:43.869506 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:43.869441 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs podName:fecbcf2f-03d7-41ad-9585-3d9193c16bc0 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:15.869428579 +0000 UTC m=+65.304413181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs") pod "network-metrics-daemon-r5pzf" (UID: "fecbcf2f-03d7-41ad-9585-3d9193c16bc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 14:30:43.969876 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:43.969833 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:43.970068 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:43.970012 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 14:30:43.970068 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:43.970034 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 14:30:43.970068 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:43.970048 2550 projected.go:196] Error preparing data for projected volume kube-api-access-z79xl for pod openshift-network-diagnostics/network-check-target-m9xxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:43.970226 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:43.970102 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl podName:9ce2c26e-1d31-4e56-a620-1a0cfb60da47 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:15.970088365 +0000 UTC m=+65.405072967 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z79xl" (UniqueName: "kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl") pod "network-check-target-m9xxq" (UID: "9ce2c26e-1d31-4e56-a620-1a0cfb60da47") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 14:30:44.114206 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:44.114125 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:44.114374 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:44.114136 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:44.114374 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:44.114247 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5pzf" podUID="fecbcf2f-03d7-41ad-9585-3d9193c16bc0" Jul 14 14:30:44.114374 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:44.114334 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9xxq" podUID="9ce2c26e-1d31-4e56-a620-1a0cfb60da47" Jul 14 14:30:44.114374 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:44.114145 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:44.114563 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:44.114435 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wrmgv" podUID="ec9d1d25-83d7-4926-abe8-191d4e3fefe7" Jul 14 14:30:45.817612 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.817540 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-194.ec2.internal" event="NodeReady" Jul 14 14:30:45.818118 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.817666 2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 14:30:45.863955 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.863856 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c8d7df94f-dqkvh"] Jul 14 14:30:45.865595 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.865573 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:45.868002 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.867979 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hzwj6"] Jul 14 14:30:45.868774 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.868754 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 14:30:45.868997 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.868978 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 14:30:45.869073 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.869014 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m9z97\"" Jul 14 14:30:45.869349 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.869332 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 14:30:45.869731 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.869713 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:45.873046 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.873025 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 14:30:45.873134 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.873098 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x4st5\"" Jul 14 14:30:45.873921 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.873901 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 14:30:45.875541 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.875436 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 14:30:45.875631 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.875585 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tsdfc"] Jul 14 14:30:45.877156 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.877136 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tsdfc" Jul 14 14:30:45.880189 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.880170 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fdbdg\"" Jul 14 14:30:45.880286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.880197 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 14:30:45.880286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.880218 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 14:30:45.880538 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.880514 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 14:30:45.883799 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.883777 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c8d7df94f-dqkvh"] Jul 14 14:30:45.890762 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.890736 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tsdfc"] Jul 14 14:30:45.899042 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.899010 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzwj6"] Jul 14 14:30:45.985337 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985297 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7a7851-c2e6-414a-9716-5296d9c8e416-config-volume\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:45.985337 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985346 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-installation-pull-secrets\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:45.985588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985462 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7a7851-c2e6-414a-9716-5296d9c8e416-tmp-dir\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:45.985588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985498 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-trusted-ca\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:45.985588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985524 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-image-registry-private-configuration\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:45.985588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985553 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsz5c\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-kube-api-access-vsz5c\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:45.985791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985619 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d24875c9-5d09-428e-a559-03ea92fbb9dd-cert\") pod \"ingress-canary-tsdfc\" (UID: \"d24875c9-5d09-428e-a559-03ea92fbb9dd\") " pod="openshift-ingress-canary/ingress-canary-tsdfc" Jul 14 14:30:45.985791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985658 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-certificates\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:45.985791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985691 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnxf\" (UniqueName: \"kubernetes.io/projected/d24875c9-5d09-428e-a559-03ea92fbb9dd-kube-api-access-tqnxf\") pod \"ingress-canary-tsdfc\" (UID: \"d24875c9-5d09-428e-a559-03ea92fbb9dd\") " pod="openshift-ingress-canary/ingress-canary-tsdfc" Jul 14 14:30:45.985791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985738 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc0a151-4810-427c-ac44-8e0656cef68e-ca-trust-extracted\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:45.985791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985775 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-bound-sa-token\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:45.986025 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985839 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:45.986025 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985866 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n292x\" (UniqueName: \"kubernetes.io/projected/bc7a7851-c2e6-414a-9716-5296d9c8e416-kube-api-access-n292x\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:45.986025 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:45.985911 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.014471 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.014421 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:46.087139 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087050 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7a7851-c2e6-414a-9716-5296d9c8e416-config-volume\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:46.087139 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087092 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-installation-pull-secrets\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.087366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087218 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7a7851-c2e6-414a-9716-5296d9c8e416-tmp-dir\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:46.087366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087251 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-trusted-ca\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.087366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087286 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-image-registry-private-configuration\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.087366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087310 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsz5c\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-kube-api-access-vsz5c\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.087366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087337 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d24875c9-5d09-428e-a559-03ea92fbb9dd-cert\") pod \"ingress-canary-tsdfc\" (UID: \"d24875c9-5d09-428e-a559-03ea92fbb9dd\") " pod="openshift-ingress-canary/ingress-canary-tsdfc" Jul 14 14:30:46.087366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087360 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-certificates\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.087649 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087386 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnxf\" (UniqueName: \"kubernetes.io/projected/d24875c9-5d09-428e-a559-03ea92fbb9dd-kube-api-access-tqnxf\") pod \"ingress-canary-tsdfc\" (UID: \"d24875c9-5d09-428e-a559-03ea92fbb9dd\") " pod="openshift-ingress-canary/ingress-canary-tsdfc" Jul 14 14:30:46.087649 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087428 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc0a151-4810-427c-ac44-8e0656cef68e-ca-trust-extracted\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.087649 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087453 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-bound-sa-token\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.087649 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087482 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:46.087649 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087509 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n292x\" (UniqueName: \"kubernetes.io/projected/bc7a7851-c2e6-414a-9716-5296d9c8e416-kube-api-access-n292x\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:46.087649 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087532 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.087649 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087603 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7a7851-c2e6-414a-9716-5296d9c8e416-tmp-dir\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:46.087990 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.087682 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 14:30:46.087990 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.087698 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8d7df94f-dqkvh: secret "image-registry-tls" not found Jul 14 14:30:46.087990 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.087718 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7a7851-c2e6-414a-9716-5296d9c8e416-config-volume\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:46.087990 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.087753 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls podName:bbc0a151-4810-427c-ac44-8e0656cef68e nodeName:}" failed. No retries permitted until 2026-07-14 14:30:46.587736186 +0000 UTC m=+36.022720803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls") pod "image-registry-7c8d7df94f-dqkvh" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e") : secret "image-registry-tls" not found Jul 14 14:30:46.088186 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.088077 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 14:30:46.088186 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.088126 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls podName:bc7a7851-c2e6-414a-9716-5296d9c8e416 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:46.588110131 +0000 UTC m=+36.023094749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls") pod "dns-default-hzwj6" (UID: "bc7a7851-c2e6-414a-9716-5296d9c8e416") : secret "dns-default-metrics-tls" not found Jul 14 14:30:46.088186 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.088171 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc0a151-4810-427c-ac44-8e0656cef68e-ca-trust-extracted\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.088399 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.088373 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-trusted-ca\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.088644 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.088622 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-certificates\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.091941 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.091922 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-installation-pull-secrets\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.092076 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.091975 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d24875c9-5d09-428e-a559-03ea92fbb9dd-cert\") pod \"ingress-canary-tsdfc\" (UID: \"d24875c9-5d09-428e-a559-03ea92fbb9dd\") " pod="openshift-ingress-canary/ingress-canary-tsdfc" Jul 14 14:30:46.092076 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.091952 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-image-registry-private-configuration\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.097791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.097760 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-bound-sa-token\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.097935 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.097899 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnxf\" (UniqueName: \"kubernetes.io/projected/d24875c9-5d09-428e-a559-03ea92fbb9dd-kube-api-access-tqnxf\") pod \"ingress-canary-tsdfc\" (UID: \"d24875c9-5d09-428e-a559-03ea92fbb9dd\") " pod="openshift-ingress-canary/ingress-canary-tsdfc" Jul 14 14:30:46.098865 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.098841 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n292x\" (UniqueName: \"kubernetes.io/projected/bc7a7851-c2e6-414a-9716-5296d9c8e416-kube-api-access-n292x\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:46.099020 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.098903 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsz5c\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-kube-api-access-vsz5c\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.107016 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.106940 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:46.113995 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.113954 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:30:46.114127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.113973 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:30:46.114127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.114101 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:30:46.117229 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.117207 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lqsw5\"" Jul 14 14:30:46.117443 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.117428 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 14:30:46.117533 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.117439 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 14:30:46.117730 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.117714 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 14:30:46.117809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.117738 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 14:30:46.117857 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.117814 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shzjn\"" Jul 14 14:30:46.194919 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.194881 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tsdfc" Jul 14 14:30:46.592978 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.592938 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:46.593168 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:46.592997 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:46.593168 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.593100 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 14:30:46.593168 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.593135 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 14:30:46.593168 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.593156 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8d7df94f-dqkvh: secret "image-registry-tls" not found Jul 14 14:30:46.593168 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.593167 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls podName:bc7a7851-c2e6-414a-9716-5296d9c8e416 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:47.593149781 +0000 UTC m=+37.028134390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls") pod "dns-default-hzwj6" (UID: "bc7a7851-c2e6-414a-9716-5296d9c8e416") : secret "dns-default-metrics-tls" not found Jul 14 14:30:46.593405 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:46.593223 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls podName:bbc0a151-4810-427c-ac44-8e0656cef68e nodeName:}" failed. No retries permitted until 2026-07-14 14:30:47.5932048 +0000 UTC m=+37.028189408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls") pod "image-registry-7c8d7df94f-dqkvh" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e") : secret "image-registry-tls" not found Jul 14 14:30:47.173321 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:47.173208 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:47.493758 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:47.493731 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tsdfc"] Jul 14 14:30:47.497778 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:30:47.497739 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24875c9_5d09_428e_a559_03ea92fbb9dd.slice/crio-76d192787cc3059a2a173625acbe8df06e4d774d44de667e15d6a78e335fb5a5 WatchSource:0}: Error finding container 76d192787cc3059a2a173625acbe8df06e4d774d44de667e15d6a78e335fb5a5: Status 404 returned error can't find the container with id 76d192787cc3059a2a173625acbe8df06e4d774d44de667e15d6a78e335fb5a5 Jul 14 14:30:47.602030 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:47.601990 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:47.602030 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:47.602029 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:47.602257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:47.602145 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 14:30:47.602257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:47.602172 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 14:30:47.602257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:47.602184 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8d7df94f-dqkvh: secret "image-registry-tls" not found Jul 14 14:30:47.602257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:47.602216 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls podName:bc7a7851-c2e6-414a-9716-5296d9c8e416 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:49.602198143 +0000 UTC m=+39.037182768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls") pod "dns-default-hzwj6" (UID: "bc7a7851-c2e6-414a-9716-5296d9c8e416") : secret "dns-default-metrics-tls" not found Jul 14 14:30:47.602257 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:47.602235 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls podName:bbc0a151-4810-427c-ac44-8e0656cef68e nodeName:}" failed. No retries permitted until 2026-07-14 14:30:49.602225467 +0000 UTC m=+39.037210069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls") pod "image-registry-7c8d7df94f-dqkvh" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e") : secret "image-registry-tls" not found Jul 14 14:30:47.676594 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:47.676558 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:48.107547 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:48.107503 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:48.266737 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:48.266705 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:48.303420 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:48.303387 2550 generic.go:356] "Generic (PLEG): container finished" podID="3da1224a-2d52-408c-99a2-0662186b0cab" containerID="6dab78b5668eb3f616be4f880d6ac72dbeea6ac0f36e0b6d05112a85734922b9" exitCode=0 Jul 14 14:30:48.303582 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:48.303493 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" event={"ID":"3da1224a-2d52-408c-99a2-0662186b0cab","Type":"ContainerDied","Data":"6dab78b5668eb3f616be4f880d6ac72dbeea6ac0f36e0b6d05112a85734922b9"} Jul 14 14:30:48.304776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:48.304750 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tsdfc" event={"ID":"d24875c9-5d09-428e-a559-03ea92fbb9dd","Type":"ContainerStarted","Data":"76d192787cc3059a2a173625acbe8df06e4d774d44de667e15d6a78e335fb5a5"} Jul 14 14:30:48.340722 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:48.340686 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:49.141489 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:49.141420 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:49.310691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:49.310648 2550 generic.go:356] "Generic (PLEG): container finished" podID="3da1224a-2d52-408c-99a2-0662186b0cab" containerID="daa19cc18651e30a2a23437483a1fbfb5064222a1c00f345c2e8042164188a77" exitCode=0 Jul 14 14:30:49.311144 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:49.310733 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" event={"ID":"3da1224a-2d52-408c-99a2-0662186b0cab","Type":"ContainerDied","Data":"daa19cc18651e30a2a23437483a1fbfb5064222a1c00f345c2e8042164188a77"} Jul 14 14:30:49.621711 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:49.621670 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:49.621910 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:49.621719 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:49.621910 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:49.621843 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 14:30:49.622041 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:49.621846 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 14:30:49.622041 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:49.621921 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls podName:bc7a7851-c2e6-414a-9716-5296d9c8e416 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:53.621900851 +0000 UTC m=+43.056885473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls") pod "dns-default-hzwj6" (UID: "bc7a7851-c2e6-414a-9716-5296d9c8e416") : secret "dns-default-metrics-tls" not found Jul 14 14:30:49.622041 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:49.621929 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8d7df94f-dqkvh: secret "image-registry-tls" not found Jul 14 14:30:49.622041 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:49.621995 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls podName:bbc0a151-4810-427c-ac44-8e0656cef68e nodeName:}" failed. No retries permitted until 2026-07-14 14:30:53.621982355 +0000 UTC m=+43.056966971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls") pod "image-registry-7c8d7df94f-dqkvh" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e") : secret "image-registry-tls" not found Jul 14 14:30:50.316110 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:50.316075 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" event={"ID":"3da1224a-2d52-408c-99a2-0662186b0cab","Type":"ContainerStarted","Data":"a3d54f43270f0d364c60612e0c7a07fb398f9031bb0aca78c7ba152dffd5daa3"} Jul 14 14:30:50.342201 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:50.342155 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dkvx6" podStartSLOduration=6.905584477 podStartE2EDuration="38.342138301s" totalImagesPullingTime="31.436553824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:50.340809268 +0000 UTC m=+39.775793895" watchObservedRunningTime="2026-07-14 14:30:50.342138301 +0000 UTC m=+39.777122966" Jul 14 14:30:50.465369 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:50.465037 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:50.569999 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:50.569888 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:50.665897 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:50.665859 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:50.766552 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:50.766514 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:30:51.322611 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:51.322574 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tsdfc" event={"ID":"d24875c9-5d09-428e-a559-03ea92fbb9dd","Type":"ContainerStarted","Data":"352dc03c25534d42924c8ea41be8c160151ec509442aee79a34ddac688cbff46"} Jul 14 14:30:51.337800 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:51.337750 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tsdfc" podStartSLOduration=3.481070734 podStartE2EDuration="6.337735156s" totalImagesPullingTime="2.856664422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:51.337502152 +0000 UTC m=+40.772486775" watchObservedRunningTime="2026-07-14 14:30:51.337735156 +0000 UTC m=+40.772719758" Jul 14 14:30:53.656348 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:53.656301 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:30:53.656348 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:53.656346 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:30:53.656782 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:53.656445 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 14:30:53.656782 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:53.656449 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 14:30:53.656782 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:53.656533 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls podName:bc7a7851-c2e6-414a-9716-5296d9c8e416 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:01.656516664 +0000 UTC m=+51.091501266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls") pod "dns-default-hzwj6" (UID: "bc7a7851-c2e6-414a-9716-5296d9c8e416") : secret "dns-default-metrics-tls" not found Jul 14 14:30:53.656782 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:53.656455 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8d7df94f-dqkvh: secret "image-registry-tls" not found Jul 14 14:30:53.656782 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:30:53.656583 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls podName:bbc0a151-4810-427c-ac44-8e0656cef68e nodeName:}" failed. No retries permitted until 2026-07-14 14:31:01.656572106 +0000 UTC m=+51.091556708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls") pod "image-registry-7c8d7df94f-dqkvh" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e") : secret "image-registry-tls" not found Jul 14 14:30:58.403876 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:30:58.403836 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:00.718537 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:00.718498 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:01.711937 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:01.711890 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:31:01.711937 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:01.711937 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:31:01.712162 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:01.712071 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 14:31:01.712162 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:01.712140 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls podName:bc7a7851-c2e6-414a-9716-5296d9c8e416 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:17.712126205 +0000 UTC m=+67.147110810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls") pod "dns-default-hzwj6" (UID: "bc7a7851-c2e6-414a-9716-5296d9c8e416") : secret "dns-default-metrics-tls" not found Jul 14 14:31:01.712236 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:01.712074 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 14:31:01.712236 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:01.712179 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8d7df94f-dqkvh: secret "image-registry-tls" not found Jul 14 14:31:01.712236 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:01.712227 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls podName:bbc0a151-4810-427c-ac44-8e0656cef68e nodeName:}" failed. No retries permitted until 2026-07-14 14:31:17.712214753 +0000 UTC m=+67.147199373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls") pod "image-registry-7c8d7df94f-dqkvh" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e") : secret "image-registry-tls" not found Jul 14 14:31:04.687940 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:04.687896 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:04.780134 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:04.780095 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:05.410397 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:05.410366 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:06.043823 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:06.043785 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:06.101859 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:06.101824 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:06.380722 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:06.380634 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:06.444054 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:06.444019 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:07.357518 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:07.357482 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:07.458329 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:07.458300 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:12.299643 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:12.299604 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6998d" Jul 14 14:31:12.453204 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:12.453163 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:12.515980 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:12.515928 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:15.915129 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:15.915074 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:31:15.915129 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:15.915140 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:31:15.918132 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:15.918107 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 14:31:15.918261 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:15.918144 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 14:31:15.926204 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:15.926184 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 14:31:15.926266 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:15.926257 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs podName:fecbcf2f-03d7-41ad-9585-3d9193c16bc0 nodeName:}" failed. No retries permitted until 2026-07-14 14:32:19.926240112 +0000 UTC m=+129.361224717 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs") pod "network-metrics-daemon-r5pzf" (UID: "fecbcf2f-03d7-41ad-9585-3d9193c16bc0") : secret "metrics-daemon-secret" not found Jul 14 14:31:15.929777 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:15.929755 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9d1d25-83d7-4926-abe8-191d4e3fefe7-original-pull-secret\") pod \"global-pull-secret-syncer-wrmgv\" (UID: \"ec9d1d25-83d7-4926-abe8-191d4e3fefe7\") " pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:31:16.016459 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.016412 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:31:16.019786 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.019766 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 14:31:16.030184 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.030153 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 14:31:16.040631 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.040608 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79xl\" (UniqueName: \"kubernetes.io/projected/9ce2c26e-1d31-4e56-a620-1a0cfb60da47-kube-api-access-z79xl\") pod \"network-check-target-m9xxq\" (UID: \"9ce2c26e-1d31-4e56-a620-1a0cfb60da47\") " pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:31:16.128146 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.128118 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lqsw5\"" Jul 14 14:31:16.135942 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.135922 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:31:16.138251 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.138235 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wrmgv" Jul 14 14:31:16.285717 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.285682 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wrmgv"] Jul 14 14:31:16.290512 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:31:16.290485 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9d1d25_83d7_4926_abe8_191d4e3fefe7.slice/crio-1122e597f8dfed3e56da54a1ad343f5632e504f3b294a6ce914f16480430172e WatchSource:0}: Error finding container 1122e597f8dfed3e56da54a1ad343f5632e504f3b294a6ce914f16480430172e: Status 404 returned error can't find the container with id 1122e597f8dfed3e56da54a1ad343f5632e504f3b294a6ce914f16480430172e Jul 14 14:31:16.300833 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.300811 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m9xxq"] Jul 14 14:31:16.304004 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:31:16.303984 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce2c26e_1d31_4e56_a620_1a0cfb60da47.slice/crio-08c65d2df8109cf070a2c162d269679a0767b4c3c3c2739965bdf47b71445754 WatchSource:0}: Error finding container 08c65d2df8109cf070a2c162d269679a0767b4c3c3c2739965bdf47b71445754: Status 404 returned error can't find the container with id 08c65d2df8109cf070a2c162d269679a0767b4c3c3c2739965bdf47b71445754 Jul 14 14:31:16.383741 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.383705 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wrmgv" event={"ID":"ec9d1d25-83d7-4926-abe8-191d4e3fefe7","Type":"ContainerStarted","Data":"1122e597f8dfed3e56da54a1ad343f5632e504f3b294a6ce914f16480430172e"} Jul 14 14:31:16.384818 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:16.384790 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m9xxq" event={"ID":"9ce2c26e-1d31-4e56-a620-1a0cfb60da47","Type":"ContainerStarted","Data":"08c65d2df8109cf070a2c162d269679a0767b4c3c3c2739965bdf47b71445754"} Jul 14 14:31:17.734972 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:17.734921 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:31:17.734972 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:17.734980 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:31:17.735421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:17.735084 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 14:31:17.735421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:17.735142 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls podName:bc7a7851-c2e6-414a-9716-5296d9c8e416 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:49.735128142 +0000 UTC m=+99.170112748 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls") pod "dns-default-hzwj6" (UID: "bc7a7851-c2e6-414a-9716-5296d9c8e416") : secret "dns-default-metrics-tls" not found Jul 14 14:31:17.735421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:17.735088 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 14:31:17.735421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:17.735165 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8d7df94f-dqkvh: secret "image-registry-tls" not found Jul 14 14:31:17.735421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:17.735194 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls podName:bbc0a151-4810-427c-ac44-8e0656cef68e nodeName:}" failed. No retries permitted until 2026-07-14 14:31:49.735185122 +0000 UTC m=+99.170169737 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls") pod "image-registry-7c8d7df94f-dqkvh" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e") : secret "image-registry-tls" not found Jul 14 14:31:20.399625 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:20.399587 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m9xxq" event={"ID":"9ce2c26e-1d31-4e56-a620-1a0cfb60da47","Type":"ContainerStarted","Data":"26d2a8c3203c333ab6d132314c4a70e77dfeac6b85fa8c261757528efe89fa62"} Jul 14 14:31:20.400117 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:20.399719 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:31:20.415626 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:20.415546 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m9xxq" podStartSLOduration=65.151728096 podStartE2EDuration="1m8.415526796s" totalImagesPullingTime="3.2637987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:20.415334586 +0000 UTC m=+69.850319211" watchObservedRunningTime="2026-07-14 14:31:20.415526796 +0000 UTC m=+69.850511421" Jul 14 14:31:22.405763 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:22.405727 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wrmgv" event={"ID":"ec9d1d25-83d7-4926-abe8-191d4e3fefe7","Type":"ContainerStarted","Data":"a20515e13f11c77e36c20aee46303113ac797cccb1bb47def1dbd8967f400bbe"} Jul 14 14:31:22.421773 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:22.421725 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wrmgv" podStartSLOduration=65.342303921 podStartE2EDuration="1m10.421710275s" totalImagesPullingTime="5.079406354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:22.421180151 +0000 UTC m=+71.856164775" watchObservedRunningTime="2026-07-14 14:31:22.421710275 +0000 UTC m=+71.856694898" Jul 14 14:31:26.227754 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:26.227716 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:26.281564 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:26.281530 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:27.950355 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:27.950318 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:28.048172 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:28.048143 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" limit=1024 Jul 14 14:31:39.504301 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.504268 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xs4rs"] Jul 14 14:31:39.509007 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.508985 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.512283 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.512262 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 14:31:39.512407 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.512346 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 14:31:39.513642 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.513622 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 14:31:39.513642 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.513634 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-chg99\"" Jul 14 14:31:39.513814 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.513625 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 14:31:39.521351 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.521331 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xs4rs"] Jul 14 14:31:39.597660 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.597618 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhsx5\" (UniqueName: \"kubernetes.io/projected/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-api-access-rhsx5\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.597660 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.597660 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.598007 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.597683 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.598007 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.597745 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/743dbe3e-0b78-4729-9eea-58edd00051e3-data-volume\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.598007 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.597868 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/743dbe3e-0b78-4729-9eea-58edd00051e3-crio-socket\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.698998 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.698940 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/743dbe3e-0b78-4729-9eea-58edd00051e3-data-volume\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.699071 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.699050 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/743dbe3e-0b78-4729-9eea-58edd00051e3-crio-socket\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.699118 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.699103 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsx5\" (UniqueName: \"kubernetes.io/projected/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-api-access-rhsx5\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.699152 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.699137 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.699187 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.699163 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.699187 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.699171 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/743dbe3e-0b78-4729-9eea-58edd00051e3-crio-socket\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.699261 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:39.699247 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 14:31:39.699297 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:39.699279 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 14:31:39.699353 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.699333 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/743dbe3e-0b78-4729-9eea-58edd00051e3-data-volume\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:39.699386 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:39.699348 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:40.199327011 +0000 UTC m=+89.634311628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : configmap "kube-rbac-proxy" not found Jul 14 14:31:39.699425 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:39.699386 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:40.199372943 +0000 UTC m=+89.634357558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : secret "insights-runtime-extractor-tls" not found Jul 14 14:31:39.710004 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:39.709956 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhsx5\" (UniqueName: \"kubernetes.io/projected/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-api-access-rhsx5\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:40.203776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:40.203736 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:40.203776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:40.203775 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:40.204020 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:40.203894 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 14:31:40.204020 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:40.203899 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 14:31:40.204020 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:40.203953 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:41.203938515 +0000 UTC m=+90.638923120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : configmap "kube-rbac-proxy" not found Jul 14 14:31:40.204020 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:40.203986 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:41.203979235 +0000 UTC m=+90.638963837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : secret "insights-runtime-extractor-tls" not found Jul 14 14:31:41.211883 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:41.211849 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:41.211883 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:41.211889 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:41.212421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:41.212024 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 14:31:41.212421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:41.212031 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 14:31:41.212421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:41.212108 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:43.212088151 +0000 UTC m=+92.647072767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : secret "insights-runtime-extractor-tls" not found Jul 14 14:31:41.212421 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:41.212181 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:43.212157287 +0000 UTC m=+92.647141895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : configmap "kube-rbac-proxy" not found Jul 14 14:31:41.303525 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:41.303500 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w4pnw_c078bb29-1f4d-45f5-beed-4d91e9a4a288/dns-node-resolver/0.log" Jul 14 14:31:41.904205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:41.904150 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mrt4j_6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a/node-ca/0.log" Jul 14 14:31:42.504296 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:42.504271 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tsdfc_d24875c9-5d09-428e-a559-03ea92fbb9dd/serve-healthcheck-canary/0.log" Jul 14 14:31:43.228729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:43.228691 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:43.228916 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:43.228783 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:43.228916 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:43.228841 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 14:31:43.228916 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:43.228871 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 14:31:43.228916 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:43.228909 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:47.228892 +0000 UTC m=+96.663876605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : configmap "kube-rbac-proxy" not found Jul 14 14:31:43.229090 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:43.228923 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:47.228917015 +0000 UTC m=+96.663901617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : secret "insights-runtime-extractor-tls" not found Jul 14 14:31:44.355933 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.355897 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn"] Jul 14 14:31:44.358791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.358770 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn" Jul 14 14:31:44.362607 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.362573 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 14:31:44.362741 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.362652 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-m8pln\"" Jul 14 14:31:44.364083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.364059 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 14:31:44.372329 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.372299 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn"] Jul 14 14:31:44.437787 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.437744 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmthq\" (UniqueName: \"kubernetes.io/projected/2d287354-d90b-40e8-9b59-0a88c906081b-kube-api-access-rmthq\") pod \"volume-data-source-validator-78495597f6-w6pbn\" (UID: \"2d287354-d90b-40e8-9b59-0a88c906081b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn" Jul 14 14:31:44.538408 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.538355 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmthq\" (UniqueName: \"kubernetes.io/projected/2d287354-d90b-40e8-9b59-0a88c906081b-kube-api-access-rmthq\") pod \"volume-data-source-validator-78495597f6-w6pbn\" (UID: \"2d287354-d90b-40e8-9b59-0a88c906081b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn" Jul 14 14:31:44.547406 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.547377 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmthq\" (UniqueName: \"kubernetes.io/projected/2d287354-d90b-40e8-9b59-0a88c906081b-kube-api-access-rmthq\") pod \"volume-data-source-validator-78495597f6-w6pbn\" (UID: \"2d287354-d90b-40e8-9b59-0a88c906081b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn" Jul 14 14:31:44.667908 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.667802 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn" Jul 14 14:31:44.792680 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:44.792647 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn"] Jul 14 14:31:44.795714 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:31:44.795688 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d287354_d90b_40e8_9b59_0a88c906081b.slice/crio-e8d228ba0c30fd0c73b17e267887862f986728262aaf00ee76b017310e6d623b WatchSource:0}: Error finding container e8d228ba0c30fd0c73b17e267887862f986728262aaf00ee76b017310e6d623b: Status 404 returned error can't find the container with id e8d228ba0c30fd0c73b17e267887862f986728262aaf00ee76b017310e6d623b Jul 14 14:31:45.465865 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:45.465834 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn" event={"ID":"2d287354-d90b-40e8-9b59-0a88c906081b","Type":"ContainerStarted","Data":"e8d228ba0c30fd0c73b17e267887862f986728262aaf00ee76b017310e6d623b"} Jul 14 14:31:46.469915 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:46.469831 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn" event={"ID":"2d287354-d90b-40e8-9b59-0a88c906081b","Type":"ContainerStarted","Data":"c1d2b0b92b06fb725b0b98fcb139335a84faa774f6a5f1cb35438ec4c82715fb"} Jul 14 14:31:46.487185 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:46.487131 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-w6pbn" podStartSLOduration=1.2551773929999999 podStartE2EDuration="2.487113807s" totalImagesPullingTime="1.231936414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:46.485477285 +0000 UTC m=+95.920461908" watchObservedRunningTime="2026-07-14 14:31:46.487113807 +0000 UTC m=+95.922098431" Jul 14 14:31:47.259826 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:47.259777 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:47.259826 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:47.259828 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:47.260098 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:47.259893 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 14:31:47.260098 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:47.259930 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 14:31:47.260098 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:47.259941 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:55.259926477 +0000 UTC m=+104.694911079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : configmap "kube-rbac-proxy" not found Jul 14 14:31:47.260098 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:47.260088 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:55.260069774 +0000 UTC m=+104.695054379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : secret "insights-runtime-extractor-tls" not found Jul 14 14:31:49.781555 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:49.781514 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:31:49.781934 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:49.781581 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") pod \"image-registry-7c8d7df94f-dqkvh\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:31:49.781934 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:49.781684 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 14:31:49.781934 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:49.781763 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls podName:bc7a7851-c2e6-414a-9716-5296d9c8e416 nodeName:}" failed. No retries permitted until 2026-07-14 14:32:53.781746585 +0000 UTC m=+163.216731186 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls") pod "dns-default-hzwj6" (UID: "bc7a7851-c2e6-414a-9716-5296d9c8e416") : secret "dns-default-metrics-tls" not found Jul 14 14:31:49.781934 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:49.781691 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 14:31:49.781934 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:49.781782 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c8d7df94f-dqkvh: secret "image-registry-tls" not found Jul 14 14:31:49.781934 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:49.781829 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls podName:bbc0a151-4810-427c-ac44-8e0656cef68e nodeName:}" failed. No retries permitted until 2026-07-14 14:32:53.781819162 +0000 UTC m=+163.216803765 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls") pod "image-registry-7c8d7df94f-dqkvh" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e") : secret "image-registry-tls" not found Jul 14 14:31:51.405133 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:51.405102 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m9xxq" Jul 14 14:31:52.645186 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.645155 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2pvbl"] Jul 14 14:31:52.671453 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.671415 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2pvbl"] Jul 14 14:31:52.671610 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.671537 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.674752 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.674725 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 14 14:31:52.674874 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.674827 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 14 14:31:52.675811 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.675790 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gz9bz\"" Jul 14 14:31:52.675926 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.675790 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 14 14:31:52.675926 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.675830 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 14 14:31:52.803382 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.803348 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea78f03e-86db-4af9-976d-c0eb9f747e3e-signing-cabundle\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.803585 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.803420 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea78f03e-86db-4af9-976d-c0eb9f747e3e-signing-key\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.803585 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.803473 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95rr\" (UniqueName: \"kubernetes.io/projected/ea78f03e-86db-4af9-976d-c0eb9f747e3e-kube-api-access-b95rr\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.904231 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.904131 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95rr\" (UniqueName: \"kubernetes.io/projected/ea78f03e-86db-4af9-976d-c0eb9f747e3e-kube-api-access-b95rr\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.904386 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.904270 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea78f03e-86db-4af9-976d-c0eb9f747e3e-signing-cabundle\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.904386 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.904328 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea78f03e-86db-4af9-976d-c0eb9f747e3e-signing-key\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.907726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.907698 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea78f03e-86db-4af9-976d-c0eb9f747e3e-signing-key\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.908064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.908042 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea78f03e-86db-4af9-976d-c0eb9f747e3e-signing-cabundle\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.915353 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.915330 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95rr\" (UniqueName: \"kubernetes.io/projected/ea78f03e-86db-4af9-976d-c0eb9f747e3e-kube-api-access-b95rr\") pod \"service-ca-79f68b9f4f-2pvbl\" (UID: \"ea78f03e-86db-4af9-976d-c0eb9f747e3e\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:52.980453 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:52.980418 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" Jul 14 14:31:53.122567 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:53.122534 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2pvbl"] Jul 14 14:31:53.125714 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:31:53.125686 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea78f03e_86db_4af9_976d_c0eb9f747e3e.slice/crio-15fdf889e4ea5b4fc4ca0984a14b1c07d4cc2059bc4c258bd38d040461be2e63 WatchSource:0}: Error finding container 15fdf889e4ea5b4fc4ca0984a14b1c07d4cc2059bc4c258bd38d040461be2e63: Status 404 returned error can't find the container with id 15fdf889e4ea5b4fc4ca0984a14b1c07d4cc2059bc4c258bd38d040461be2e63 Jul 14 14:31:53.487759 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:53.487721 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" event={"ID":"ea78f03e-86db-4af9-976d-c0eb9f747e3e","Type":"ContainerStarted","Data":"15fdf889e4ea5b4fc4ca0984a14b1c07d4cc2059bc4c258bd38d040461be2e63"} Jul 14 14:31:55.324211 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:55.324171 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:55.324211 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:55.324219 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:55.324748 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:55.324352 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 14:31:55.324748 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:31:55.324428 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls podName:743dbe3e-0b78-4729-9eea-58edd00051e3 nodeName:}" failed. No retries permitted until 2026-07-14 14:32:11.324410211 +0000 UTC m=+120.759394816 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xs4rs" (UID: "743dbe3e-0b78-4729-9eea-58edd00051e3") : secret "insights-runtime-extractor-tls" not found Jul 14 14:31:55.324915 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:55.324889 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/743dbe3e-0b78-4729-9eea-58edd00051e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:31:56.499653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:56.499568 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" event={"ID":"ea78f03e-86db-4af9-976d-c0eb9f747e3e","Type":"ContainerStarted","Data":"9f366c7a256eab54edb952905bbf8acc6aa3d87debc1c4d0494440cbdb1d4403"} Jul 14 14:31:56.516210 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:31:56.516159 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-2pvbl" podStartSLOduration=1.59304194 podStartE2EDuration="4.516143646s" totalImagesPullingTime="2.923101706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:56.514406887 +0000 UTC m=+105.949391512" watchObservedRunningTime="2026-07-14 14:31:56.516143646 +0000 UTC m=+105.951128270" Jul 14 14:32:11.347265 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:11.347224 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:32:11.349634 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:11.349609 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/743dbe3e-0b78-4729-9eea-58edd00051e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xs4rs\" (UID: \"743dbe3e-0b78-4729-9eea-58edd00051e3\") " pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:32:11.620316 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:11.620227 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-chg99\"" Jul 14 14:32:11.627664 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:11.627636 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xs4rs" Jul 14 14:32:11.755998 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:11.755948 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xs4rs"] Jul 14 14:32:11.759687 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:11.759662 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743dbe3e_0b78_4729_9eea_58edd00051e3.slice/crio-a653522f737dd24d9fc8f72c6719524b3f3d4bcc4d3b232302d8e55eece45e9b WatchSource:0}: Error finding container a653522f737dd24d9fc8f72c6719524b3f3d4bcc4d3b232302d8e55eece45e9b: Status 404 returned error can't find the container with id a653522f737dd24d9fc8f72c6719524b3f3d4bcc4d3b232302d8e55eece45e9b Jul 14 14:32:12.561312 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:12.561278 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xs4rs" event={"ID":"743dbe3e-0b78-4729-9eea-58edd00051e3","Type":"ContainerStarted","Data":"a653522f737dd24d9fc8f72c6719524b3f3d4bcc4d3b232302d8e55eece45e9b"} Jul 14 14:32:13.451231 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.451193 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf"] Jul 14 14:32:13.454429 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.453351 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf" Jul 14 14:32:13.455138 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.454698 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs"] Jul 14 14:32:13.456415 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.456334 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7pgcq\"" Jul 14 14:32:13.456896 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.456876 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.459827 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.459804 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 14 14:32:13.460069 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.460047 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 14:32:13.460358 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.460227 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 14 14:32:13.461193 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.461008 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 14:32:13.462776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.462755 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 14:32:13.463008 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.462992 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 14 14:32:13.466753 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.466032 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 14 14:32:13.467542 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.467519 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf"] Jul 14 14:32:13.474489 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.474457 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs"] Jul 14 14:32:13.564754 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.564722 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.564754 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.564756 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/db775aa7-920e-494b-9733-2c4d5311f0e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.565319 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.564788 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224fw\" (UniqueName: \"kubernetes.io/projected/f3473393-4352-4477-b304-4a3d95a03898-kube-api-access-224fw\") pod \"network-check-source-869dcfcdcd-knxlf\" (UID: \"f3473393-4352-4477-b304-4a3d95a03898\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf" Jul 14 14:32:13.565319 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.564845 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdhk\" (UniqueName: \"kubernetes.io/projected/db775aa7-920e-494b-9733-2c4d5311f0e4-kube-api-access-7rdhk\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.565319 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.564877 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-ca\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.565319 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.565007 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-hub\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.565319 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.565037 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.665535 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.665444 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/db775aa7-920e-494b-9733-2c4d5311f0e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.666035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.665498 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224fw\" (UniqueName: \"kubernetes.io/projected/f3473393-4352-4477-b304-4a3d95a03898-kube-api-access-224fw\") pod \"network-check-source-869dcfcdcd-knxlf\" (UID: \"f3473393-4352-4477-b304-4a3d95a03898\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf" Jul 14 14:32:13.666035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.665819 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdhk\" (UniqueName: \"kubernetes.io/projected/db775aa7-920e-494b-9733-2c4d5311f0e4-kube-api-access-7rdhk\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.666035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.665857 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-ca\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.666035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.665931 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-hub\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.666305 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.666272 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/db775aa7-920e-494b-9733-2c4d5311f0e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.666388 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.666358 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.666462 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.666416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.668888 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.668866 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-ca\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.669316 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.669272 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.669316 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.669284 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-hub\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.669471 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.669327 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db775aa7-920e-494b-9733-2c4d5311f0e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.684697 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.684667 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-224fw\" (UniqueName: \"kubernetes.io/projected/f3473393-4352-4477-b304-4a3d95a03898-kube-api-access-224fw\") pod \"network-check-source-869dcfcdcd-knxlf\" (UID: \"f3473393-4352-4477-b304-4a3d95a03898\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf" Jul 14 14:32:13.687952 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.687916 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdhk\" (UniqueName: \"kubernetes.io/projected/db775aa7-920e-494b-9733-2c4d5311f0e4-kube-api-access-7rdhk\") pod \"cluster-proxy-proxy-agent-75974cc666-t89gs\" (UID: \"db775aa7-920e-494b-9733-2c4d5311f0e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.767469 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.767430 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf" Jul 14 14:32:13.785149 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.785115 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" Jul 14 14:32:13.933051 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.932989 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf"] Jul 14 14:32:13.937834 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:13.937794 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3473393_4352_4477_b304_4a3d95a03898.slice/crio-5993c3b8eb96223a1c4dd0e4d8b6b5a60f66914ece92664802a47fdb3c97b243 WatchSource:0}: Error finding container 5993c3b8eb96223a1c4dd0e4d8b6b5a60f66914ece92664802a47fdb3c97b243: Status 404 returned error can't find the container with id 5993c3b8eb96223a1c4dd0e4d8b6b5a60f66914ece92664802a47fdb3c97b243 Jul 14 14:32:13.960688 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:13.960656 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs"] Jul 14 14:32:13.966043 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:13.966004 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb775aa7_920e_494b_9733_2c4d5311f0e4.slice/crio-31d9548f012cbaa96e596f4486081ce744536c82929ecd94f4c39e4dad34b8b3 WatchSource:0}: Error finding container 31d9548f012cbaa96e596f4486081ce744536c82929ecd94f4c39e4dad34b8b3: Status 404 returned error can't find the container with id 31d9548f012cbaa96e596f4486081ce744536c82929ecd94f4c39e4dad34b8b3 Jul 14 14:32:14.568863 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:14.568822 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf" event={"ID":"f3473393-4352-4477-b304-4a3d95a03898","Type":"ContainerStarted","Data":"0ad5f5fba6ac844c08beae50721696c9f09865b55d1ea4f6732f95f4b9b29b59"} Jul 14 14:32:14.569361 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:14.568887 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf" event={"ID":"f3473393-4352-4477-b304-4a3d95a03898","Type":"ContainerStarted","Data":"5993c3b8eb96223a1c4dd0e4d8b6b5a60f66914ece92664802a47fdb3c97b243"} Jul 14 14:32:14.570113 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:14.570082 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" event={"ID":"db775aa7-920e-494b-9733-2c4d5311f0e4","Type":"ContainerStarted","Data":"31d9548f012cbaa96e596f4486081ce744536c82929ecd94f4c39e4dad34b8b3"} Jul 14 14:32:14.584486 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:14.584423 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-knxlf" podStartSLOduration=1.584403085 podStartE2EDuration="1.584403085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:14.584297134 +0000 UTC m=+124.019281760" watchObservedRunningTime="2026-07-14 14:32:14.584403085 +0000 UTC m=+124.019387711" Jul 14 14:32:17.590518 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:17.590485 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xs4rs" event={"ID":"743dbe3e-0b78-4729-9eea-58edd00051e3","Type":"ContainerStarted","Data":"268b7a5d3c1d629283d5fe819bd83cff45b932f5dc94d26778bf00929e548ed2"} Jul 14 14:32:19.600847 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:19.600798 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xs4rs" event={"ID":"743dbe3e-0b78-4729-9eea-58edd00051e3","Type":"ContainerStarted","Data":"c5ec24d86a27d87ceab3c6c58f3c4985f445fe065d42ea2ce4a2be50ee99f9a6"} Jul 14 14:32:19.603151 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:19.603120 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" event={"ID":"db775aa7-920e-494b-9733-2c4d5311f0e4","Type":"ContainerStarted","Data":"e97a6bad89c6017c13ae8c9fce8721a64b461809459a085057712ad69986302f"} Jul 14 14:32:19.603303 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:19.603171 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" event={"ID":"db775aa7-920e-494b-9733-2c4d5311f0e4","Type":"ContainerStarted","Data":"e3ddae8381afc2db29f108895e9437163b7fe4f5aed549144b210ce75406d1b5"} Jul 14 14:32:19.603303 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:19.603188 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" event={"ID":"db775aa7-920e-494b-9733-2c4d5311f0e4","Type":"ContainerStarted","Data":"cbdcefac8ffc8915912890f1cbcb9d341115ca0ea6d6e2c969055103183a770c"} Jul 14 14:32:19.623517 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:19.623452 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75974cc666-t89gs" podStartSLOduration=1.7360524229999998 podStartE2EDuration="6.623430749s" totalImagesPullingTime="4.887378326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:19.621201847 +0000 UTC m=+129.056186472" watchObservedRunningTime="2026-07-14 14:32:19.623430749 +0000 UTC m=+129.058415375" Jul 14 14:32:20.025313 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:20.025278 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:32:20.027977 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:20.027939 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecbcf2f-03d7-41ad-9585-3d9193c16bc0-metrics-certs\") pod \"network-metrics-daemon-r5pzf\" (UID: \"fecbcf2f-03d7-41ad-9585-3d9193c16bc0\") " pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:32:20.035763 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:20.035736 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shzjn\"" Jul 14 14:32:20.043301 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:20.043277 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5pzf" Jul 14 14:32:20.191271 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:20.191238 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5pzf"] Jul 14 14:32:20.194637 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:20.194599 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecbcf2f_03d7_41ad_9585_3d9193c16bc0.slice/crio-e6bf0099d92e8938b0be4a92db664a25a0d8798876a1e2e0f517d772dc46d835 WatchSource:0}: Error finding container e6bf0099d92e8938b0be4a92db664a25a0d8798876a1e2e0f517d772dc46d835: Status 404 returned error can't find the container with id e6bf0099d92e8938b0be4a92db664a25a0d8798876a1e2e0f517d772dc46d835 Jul 14 14:32:20.609420 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:20.609369 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5pzf" event={"ID":"fecbcf2f-03d7-41ad-9585-3d9193c16bc0","Type":"ContainerStarted","Data":"e6bf0099d92e8938b0be4a92db664a25a0d8798876a1e2e0f517d772dc46d835"} Jul 14 14:32:22.617949 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:22.617906 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5pzf" event={"ID":"fecbcf2f-03d7-41ad-9585-3d9193c16bc0","Type":"ContainerStarted","Data":"e5b16c18cdd76e91f241279dc0d76d47e0078b34dc0e6e1864dcd08629827bc7"} Jul 14 14:32:22.618419 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:22.617985 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5pzf" event={"ID":"fecbcf2f-03d7-41ad-9585-3d9193c16bc0","Type":"ContainerStarted","Data":"55829acef75cdb415351f55f70dff99bdb9b46da201de0aba20ede49de14d228"} Jul 14 14:32:22.620017 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:22.619989 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xs4rs" event={"ID":"743dbe3e-0b78-4729-9eea-58edd00051e3","Type":"ContainerStarted","Data":"de477891975fa3a21a454f4d6777e8ce8e5cff69c32dc5fa581f7988e81511fe"} Jul 14 14:32:22.634972 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:22.634906 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r5pzf" podStartSLOduration=128.740899864 podStartE2EDuration="2m10.634891579s" totalImagesPullingTime="1.893991715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:22.633029671 +0000 UTC m=+132.068014294" watchObservedRunningTime="2026-07-14 14:32:22.634891579 +0000 UTC m=+132.069876203" Jul 14 14:32:22.650521 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:22.650467 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xs4rs" podStartSLOduration=33.962397357 podStartE2EDuration="43.650450999s" totalImagesPullingTime="9.688053642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:31:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:22.649662532 +0000 UTC m=+132.084647158" watchObservedRunningTime="2026-07-14 14:32:22.650450999 +0000 UTC m=+132.085435623" Jul 14 14:32:26.019205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.019168 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-74mth"] Jul 14 14:32:26.021908 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.021886 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.024948 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.024922 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 14:32:26.025240 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.025219 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 14:32:26.026259 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.026238 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 14:32:26.026395 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.026356 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gfn6m\"" Jul 14 14:32:26.026525 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.026286 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 14:32:26.026525 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.026295 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 14:32:26.026643 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.026286 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 14:32:26.174531 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174485 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-wtmp\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.174531 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174536 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgwl9\" (UniqueName: \"kubernetes.io/projected/0dec370b-da14-40f6-8ffe-6c05d5321e26-kube-api-access-bgwl9\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.174793 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174605 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-sys\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.174793 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174695 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.174793 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174748 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-tls\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.174919 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174820 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-root\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.174919 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174875 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dec370b-da14-40f6-8ffe-6c05d5321e26-metrics-client-ca\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.175021 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174918 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-accelerators-collector-config\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.175021 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.174952 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-textfile\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276401 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276297 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-root\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276401 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276354 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dec370b-da14-40f6-8ffe-6c05d5321e26-metrics-client-ca\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276401 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276384 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-accelerators-collector-config\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276414 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-textfile\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276418 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-root\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276460 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-wtmp\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276501 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgwl9\" (UniqueName: \"kubernetes.io/projected/0dec370b-da14-40f6-8ffe-6c05d5321e26-kube-api-access-bgwl9\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276541 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-sys\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276567 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276596 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-tls\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.276691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276669 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-sys\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.277110 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276786 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-textfile\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.277110 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.276789 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-wtmp\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.277225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.277202 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dec370b-da14-40f6-8ffe-6c05d5321e26-metrics-client-ca\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.277282 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.277202 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-accelerators-collector-config\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.279225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.279191 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.279225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.279212 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dec370b-da14-40f6-8ffe-6c05d5321e26-node-exporter-tls\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.287502 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.287472 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgwl9\" (UniqueName: \"kubernetes.io/projected/0dec370b-da14-40f6-8ffe-6c05d5321e26-kube-api-access-bgwl9\") pod \"node-exporter-74mth\" (UID: \"0dec370b-da14-40f6-8ffe-6c05d5321e26\") " pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.333511 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.333474 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74mth" Jul 14 14:32:26.343296 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:26.343269 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dec370b_da14_40f6_8ffe_6c05d5321e26.slice/crio-7e09dc9c473e68cb2441d95d48c9fd60df12a425bb028e425d4065828eb26feb WatchSource:0}: Error finding container 7e09dc9c473e68cb2441d95d48c9fd60df12a425bb028e425d4065828eb26feb: Status 404 returned error can't find the container with id 7e09dc9c473e68cb2441d95d48c9fd60df12a425bb028e425d4065828eb26feb Jul 14 14:32:26.633176 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:26.633079 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74mth" event={"ID":"0dec370b-da14-40f6-8ffe-6c05d5321e26","Type":"ContainerStarted","Data":"7e09dc9c473e68cb2441d95d48c9fd60df12a425bb028e425d4065828eb26feb"} Jul 14 14:32:27.637837 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:27.637803 2550 generic.go:356] "Generic (PLEG): container finished" podID="0dec370b-da14-40f6-8ffe-6c05d5321e26" containerID="6175079b286fac397014f103f0e27bd6c8e7b2e48f17b29d89009134a7758ac5" exitCode=0 Jul 14 14:32:27.638211 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:27.637885 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74mth" event={"ID":"0dec370b-da14-40f6-8ffe-6c05d5321e26","Type":"ContainerDied","Data":"6175079b286fac397014f103f0e27bd6c8e7b2e48f17b29d89009134a7758ac5"} Jul 14 14:32:28.643624 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:28.643587 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74mth" event={"ID":"0dec370b-da14-40f6-8ffe-6c05d5321e26","Type":"ContainerStarted","Data":"9931a72b9dcb4f01336e01d3779d1a5815070e5b9331ed23b505de0ad2890d8c"} Jul 14 14:32:28.644033 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:28.643642 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74mth" event={"ID":"0dec370b-da14-40f6-8ffe-6c05d5321e26","Type":"ContainerStarted","Data":"17c1a194acda7d657ec796e83e77335800543b4ad4a5108b25ecdd9d421aaf17"} Jul 14 14:32:28.664302 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:28.664251 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-74mth" podStartSLOduration=2.45840434 podStartE2EDuration="3.664229679s" totalImagesPullingTime="1.14146111s" totalInitContainerRuntime="64.364229ms" isStatefulPod=true podCreationTimestamp="2026-07-14 14:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:28.662450184 +0000 UTC m=+138.097434809" watchObservedRunningTime="2026-07-14 14:32:28.664229679 +0000 UTC m=+138.099214303" Jul 14 14:32:30.787936 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:30.787899 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2"] Jul 14 14:32:30.791047 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:30.791029 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:30.794522 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:30.794495 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 14 14:32:30.794664 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:30.794562 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fqtjb\"" Jul 14 14:32:30.802916 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:30.802892 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2"] Jul 14 14:32:30.919694 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:30.919647 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7b7v\" (UniqueName: \"kubernetes.io/projected/0100d8ad-e992-42e7-acad-5fb3743c8c5c-kube-api-access-s7b7v\") pod \"monitoring-plugin-5f68df7f58-kpqp2\" (UID: \"0100d8ad-e992-42e7-acad-5fb3743c8c5c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:30.919873 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:30.919711 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0100d8ad-e992-42e7-acad-5fb3743c8c5c-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-kpqp2\" (UID: \"0100d8ad-e992-42e7-acad-5fb3743c8c5c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:31.020476 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:31.020424 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7b7v\" (UniqueName: \"kubernetes.io/projected/0100d8ad-e992-42e7-acad-5fb3743c8c5c-kube-api-access-s7b7v\") pod \"monitoring-plugin-5f68df7f58-kpqp2\" (UID: \"0100d8ad-e992-42e7-acad-5fb3743c8c5c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:31.020660 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:31.020506 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0100d8ad-e992-42e7-acad-5fb3743c8c5c-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-kpqp2\" (UID: \"0100d8ad-e992-42e7-acad-5fb3743c8c5c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:31.023069 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:31.023041 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0100d8ad-e992-42e7-acad-5fb3743c8c5c-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-kpqp2\" (UID: \"0100d8ad-e992-42e7-acad-5fb3743c8c5c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:31.028903 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:31.028880 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7b7v\" (UniqueName: \"kubernetes.io/projected/0100d8ad-e992-42e7-acad-5fb3743c8c5c-kube-api-access-s7b7v\") pod \"monitoring-plugin-5f68df7f58-kpqp2\" (UID: \"0100d8ad-e992-42e7-acad-5fb3743c8c5c\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:31.100302 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:31.100220 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:31.244391 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:31.244367 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2"] Jul 14 14:32:31.246900 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:31.246869 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0100d8ad_e992_42e7_acad_5fb3743c8c5c.slice/crio-3bab97d4645ee7c00f79a6f47515f39ef9ea45ffaf18b919f1682c813f304bbf WatchSource:0}: Error finding container 3bab97d4645ee7c00f79a6f47515f39ef9ea45ffaf18b919f1682c813f304bbf: Status 404 returned error can't find the container with id 3bab97d4645ee7c00f79a6f47515f39ef9ea45ffaf18b919f1682c813f304bbf Jul 14 14:32:31.658740 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:31.658707 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" event={"ID":"0100d8ad-e992-42e7-acad-5fb3743c8c5c","Type":"ContainerStarted","Data":"3bab97d4645ee7c00f79a6f47515f39ef9ea45ffaf18b919f1682c813f304bbf"} Jul 14 14:32:33.665957 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:33.665919 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" event={"ID":"0100d8ad-e992-42e7-acad-5fb3743c8c5c","Type":"ContainerStarted","Data":"ffb4b89d4aa8eebfe8479d22305f9a89abc091d836a1dd15b07ca409229c37e0"} Jul 14 14:32:33.666373 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:33.666184 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:33.670780 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:33.670756 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" Jul 14 14:32:33.683249 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:33.683157 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-kpqp2" podStartSLOduration=1.768344496 podStartE2EDuration="3.683137752s" totalImagesPullingTime="1.914793256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:33.68251075 +0000 UTC m=+143.117495374" watchObservedRunningTime="2026-07-14 14:32:33.683137752 +0000 UTC m=+143.118122378" Jul 14 14:32:38.377318 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.377284 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c8d7df94f-dqkvh"] Jul 14 14:32:38.377684 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:32:38.377469 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" podUID="bbc0a151-4810-427c-ac44-8e0656cef68e" Jul 14 14:32:38.661879 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.661793 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-6h6nq"] Jul 14 14:32:38.665222 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.665201 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-6h6nq" Jul 14 14:32:38.668273 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.668247 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rjd96\"" Jul 14 14:32:38.668391 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.668248 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 14:32:38.668391 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.668258 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 14:32:38.678546 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.678525 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-6h6nq"] Jul 14 14:32:38.685938 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.685916 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:32:38.692078 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.692055 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:32:38.780067 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780036 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc0a151-4810-427c-ac44-8e0656cef68e-ca-trust-extracted\") pod \"bbc0a151-4810-427c-ac44-8e0656cef68e\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " Jul 14 14:32:38.780252 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780077 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsz5c\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-kube-api-access-vsz5c\") pod \"bbc0a151-4810-427c-ac44-8e0656cef68e\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " Jul 14 14:32:38.780252 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780106 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-certificates\") pod \"bbc0a151-4810-427c-ac44-8e0656cef68e\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " Jul 14 14:32:38.780252 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780138 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-image-registry-private-configuration\") pod \"bbc0a151-4810-427c-ac44-8e0656cef68e\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " Jul 14 14:32:38.780252 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780180 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-trusted-ca\") pod \"bbc0a151-4810-427c-ac44-8e0656cef68e\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " Jul 14 14:32:38.780252 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780197 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-bound-sa-token\") pod \"bbc0a151-4810-427c-ac44-8e0656cef68e\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " Jul 14 14:32:38.780252 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780230 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-installation-pull-secrets\") pod \"bbc0a151-4810-427c-ac44-8e0656cef68e\" (UID: \"bbc0a151-4810-427c-ac44-8e0656cef68e\") " Jul 14 14:32:38.780530 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780346 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqs4\" (UniqueName: \"kubernetes.io/projected/d58b7a59-b913-445f-bec8-afe6c0fb75c3-kube-api-access-7wqs4\") pod \"downloads-68c9cbb497-6h6nq\" (UID: \"d58b7a59-b913-445f-bec8-afe6c0fb75c3\") " pod="openshift-console/downloads-68c9cbb497-6h6nq" Jul 14 14:32:38.780530 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780389 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc0a151-4810-427c-ac44-8e0656cef68e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bbc0a151-4810-427c-ac44-8e0656cef68e" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:32:38.780530 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780509 2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc0a151-4810-427c-ac44-8e0656cef68e-ca-trust-extracted\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:32:38.780769 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780740 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bbc0a151-4810-427c-ac44-8e0656cef68e" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:32:38.780904 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.780792 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bbc0a151-4810-427c-ac44-8e0656cef68e" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:32:38.782644 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.782616 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bbc0a151-4810-427c-ac44-8e0656cef68e" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:32:38.782731 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.782654 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-kube-api-access-vsz5c" (OuterVolumeSpecName: "kube-api-access-vsz5c") pod "bbc0a151-4810-427c-ac44-8e0656cef68e" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e"). InnerVolumeSpecName "kube-api-access-vsz5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:32:38.782731 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.782700 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bbc0a151-4810-427c-ac44-8e0656cef68e" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:32:38.782809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.782729 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bbc0a151-4810-427c-ac44-8e0656cef68e" (UID: "bbc0a151-4810-427c-ac44-8e0656cef68e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:32:38.881647 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.881613 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqs4\" (UniqueName: \"kubernetes.io/projected/d58b7a59-b913-445f-bec8-afe6c0fb75c3-kube-api-access-7wqs4\") pod \"downloads-68c9cbb497-6h6nq\" (UID: \"d58b7a59-b913-445f-bec8-afe6c0fb75c3\") " pod="openshift-console/downloads-68c9cbb497-6h6nq" Jul 14 14:32:38.881794 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.881689 2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-image-registry-private-configuration\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:32:38.881794 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.881701 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-trusted-ca\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:32:38.881794 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.881711 2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-bound-sa-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:32:38.881794 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.881720 2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc0a151-4810-427c-ac44-8e0656cef68e-installation-pull-secrets\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:32:38.881794 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.881729 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsz5c\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-kube-api-access-vsz5c\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:32:38.881794 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.881738 2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-certificates\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:32:38.897491 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.897464 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqs4\" (UniqueName: \"kubernetes.io/projected/d58b7a59-b913-445f-bec8-afe6c0fb75c3-kube-api-access-7wqs4\") pod \"downloads-68c9cbb497-6h6nq\" (UID: \"d58b7a59-b913-445f-bec8-afe6c0fb75c3\") " pod="openshift-console/downloads-68c9cbb497-6h6nq" Jul 14 14:32:38.974555 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:38.974454 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-6h6nq" Jul 14 14:32:39.118145 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:39.118117 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58b7a59_b913_445f_bec8_afe6c0fb75c3.slice/crio-8c7b7658e6c47c7a2a74886b4e2161b121297ddf3662f61f8d5b467dabe5949e WatchSource:0}: Error finding container 8c7b7658e6c47c7a2a74886b4e2161b121297ddf3662f61f8d5b467dabe5949e: Status 404 returned error can't find the container with id 8c7b7658e6c47c7a2a74886b4e2161b121297ddf3662f61f8d5b467dabe5949e Jul 14 14:32:39.118337 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:39.118318 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-6h6nq"] Jul 14 14:32:39.690578 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:39.690546 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c8d7df94f-dqkvh" Jul 14 14:32:39.690578 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:39.690561 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-6h6nq" event={"ID":"d58b7a59-b913-445f-bec8-afe6c0fb75c3","Type":"ContainerStarted","Data":"8c7b7658e6c47c7a2a74886b4e2161b121297ddf3662f61f8d5b467dabe5949e"} Jul 14 14:32:39.738533 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:39.738498 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c8d7df94f-dqkvh"] Jul 14 14:32:39.743420 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:39.743386 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c8d7df94f-dqkvh"] Jul 14 14:32:39.889282 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:39.889241 2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc0a151-4810-427c-ac44-8e0656cef68e-registry-tls\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:32:41.119435 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:41.119403 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc0a151-4810-427c-ac44-8e0656cef68e" path="/var/lib/kubelet/pods/bbc0a151-4810-427c-ac44-8e0656cef68e/volumes" Jul 14 14:32:47.915875 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.915830 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b4b5cd577-dm8fm"] Jul 14 14:32:47.919092 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.919068 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:47.922231 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.922204 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 14:32:47.922350 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.922228 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7s2hv\"" Jul 14 14:32:47.923246 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.923224 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 14:32:47.923354 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.923262 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 14:32:47.923480 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.923460 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 14:32:47.923594 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.923577 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 14:32:47.929707 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.929682 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4b5cd577-dm8fm"] Jul 14 14:32:47.956074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.956041 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-service-ca\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:47.956074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.956077 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxxt\" (UniqueName: \"kubernetes.io/projected/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-kube-api-access-qpxxt\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:47.956305 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.956108 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-serving-cert\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:47.956305 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.956138 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-config\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:47.956305 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.956226 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-oauth-config\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:47.956305 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:47.956249 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-oauth-serving-cert\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.057381 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.057337 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-oauth-config\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.057570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.057391 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-oauth-serving-cert\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.057570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.057437 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-service-ca\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.057570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.057462 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxxt\" (UniqueName: \"kubernetes.io/projected/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-kube-api-access-qpxxt\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.057570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.057492 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-serving-cert\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.057570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.057523 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-config\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.061150 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.059032 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-service-ca\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.061150 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.059032 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-config\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.061150 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.059433 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-oauth-serving-cert\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.064891 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.064864 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-serving-cert\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.064891 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.064884 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-oauth-config\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.067463 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.067442 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxxt\" (UniqueName: \"kubernetes.io/projected/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-kube-api-access-qpxxt\") pod \"console-b4b5cd577-dm8fm\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.231125 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.231040 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:32:48.375881 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.375856 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4b5cd577-dm8fm"] Jul 14 14:32:48.378461 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:48.378429 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99a4c9d_6dce_4fa8_bfb0_cf8a30e8d3e5.slice/crio-e0d386b843bba06734cdd5c87d1a973188ffad88fff1629e747696b956d0ac07 WatchSource:0}: Error finding container e0d386b843bba06734cdd5c87d1a973188ffad88fff1629e747696b956d0ac07: Status 404 returned error can't find the container with id e0d386b843bba06734cdd5c87d1a973188ffad88fff1629e747696b956d0ac07 Jul 14 14:32:48.724935 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:48.724897 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b5cd577-dm8fm" event={"ID":"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5","Type":"ContainerStarted","Data":"e0d386b843bba06734cdd5c87d1a973188ffad88fff1629e747696b956d0ac07"} Jul 14 14:32:48.885287 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:32:48.885243 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hzwj6" podUID="bc7a7851-c2e6-414a-9716-5296d9c8e416" Jul 14 14:32:49.728323 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:49.728294 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzwj6" Jul 14 14:32:53.806264 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:53.806214 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:32:53.808868 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:53.808837 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc7a7851-c2e6-414a-9716-5296d9c8e416-metrics-tls\") pod \"dns-default-hzwj6\" (UID: \"bc7a7851-c2e6-414a-9716-5296d9c8e416\") " pod="openshift-dns/dns-default-hzwj6" Jul 14 14:32:53.932857 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:53.932826 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x4st5\"" Jul 14 14:32:53.940163 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:53.940129 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzwj6" Jul 14 14:32:55.152047 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.152006 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-794fbf756f-2s5wn"] Jul 14 14:32:55.155402 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.155371 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.164421 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.164373 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 14:32:55.166308 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.166283 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794fbf756f-2s5wn"] Jul 14 14:32:55.218533 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.218490 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-service-ca\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.218708 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.218542 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnqg7\" (UniqueName: \"kubernetes.io/projected/58d3e5d4-4947-4731-b0ff-f82ed2b82494-kube-api-access-nnqg7\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.218708 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.218626 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-oauth-config\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.218708 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.218661 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-config\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.218708 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.218689 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-oauth-serving-cert\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.218926 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.218769 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-trusted-ca-bundle\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.218926 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.218850 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-serving-cert\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.319823 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.319783 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-trusted-ca-bundle\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.319823 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.319830 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-serving-cert\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.319906 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-service-ca\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.319928 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnqg7\" (UniqueName: \"kubernetes.io/projected/58d3e5d4-4947-4731-b0ff-f82ed2b82494-kube-api-access-nnqg7\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.319950 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-oauth-config\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.319994 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-config\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.320023 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-oauth-serving-cert\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320818 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.320784 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-oauth-serving-cert\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320938 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.320785 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-service-ca\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320938 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.320848 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-config\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.320938 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.320883 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-trusted-ca-bundle\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.322940 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.322913 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-serving-cert\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.323068 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.323002 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-oauth-config\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.330675 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.330652 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnqg7\" (UniqueName: \"kubernetes.io/projected/58d3e5d4-4947-4731-b0ff-f82ed2b82494-kube-api-access-nnqg7\") pod \"console-794fbf756f-2s5wn\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:55.467623 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:55.467522 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:32:58.400514 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.400091 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzwj6"] Jul 14 14:32:58.402772 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:58.402731 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7a7851_c2e6_414a_9716_5296d9c8e416.slice/crio-1b56b999baf86cbde8e952a9f756a13081b6c4289272e7a2a63bad02fd1f6143 WatchSource:0}: Error finding container 1b56b999baf86cbde8e952a9f756a13081b6c4289272e7a2a63bad02fd1f6143: Status 404 returned error can't find the container with id 1b56b999baf86cbde8e952a9f756a13081b6c4289272e7a2a63bad02fd1f6143 Jul 14 14:32:58.421390 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.421352 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794fbf756f-2s5wn"] Jul 14 14:32:58.426331 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:32:58.426301 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d3e5d4_4947_4731_b0ff_f82ed2b82494.slice/crio-3d5ca4dfe1c2e949326db5edca1f606f606f28420c8cde5ab0f404546213d841 WatchSource:0}: Error finding container 3d5ca4dfe1c2e949326db5edca1f606f606f28420c8cde5ab0f404546213d841: Status 404 returned error can't find the container with id 3d5ca4dfe1c2e949326db5edca1f606f606f28420c8cde5ab0f404546213d841 Jul 14 14:32:58.765211 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.765172 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b5cd577-dm8fm" event={"ID":"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5","Type":"ContainerStarted","Data":"324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2"} Jul 14 14:32:58.766691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.766662 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzwj6" event={"ID":"bc7a7851-c2e6-414a-9716-5296d9c8e416","Type":"ContainerStarted","Data":"1b56b999baf86cbde8e952a9f756a13081b6c4289272e7a2a63bad02fd1f6143"} Jul 14 14:32:58.768422 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.768394 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-6h6nq" event={"ID":"d58b7a59-b913-445f-bec8-afe6c0fb75c3","Type":"ContainerStarted","Data":"0ed417e5229c28eb2f7a92ef561f0c4dc5282cf190a47dd673785d3712fb9cea"} Jul 14 14:32:58.768615 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.768548 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-6h6nq" Jul 14 14:32:58.770373 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.770306 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794fbf756f-2s5wn" event={"ID":"58d3e5d4-4947-4731-b0ff-f82ed2b82494","Type":"ContainerStarted","Data":"d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34"} Jul 14 14:32:58.770373 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.770349 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794fbf756f-2s5wn" event={"ID":"58d3e5d4-4947-4731-b0ff-f82ed2b82494","Type":"ContainerStarted","Data":"3d5ca4dfe1c2e949326db5edca1f606f606f28420c8cde5ab0f404546213d841"} Jul 14 14:32:58.782191 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.782161 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-6h6nq" Jul 14 14:32:58.782919 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.782855 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b4b5cd577-dm8fm" podStartSLOduration=1.9068376329999999 podStartE2EDuration="11.782836219s" totalImagesPullingTime="9.875998586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:58.782539315 +0000 UTC m=+168.217523957" watchObservedRunningTime="2026-07-14 14:32:58.782836219 +0000 UTC m=+168.217820845" Jul 14 14:32:58.805727 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.805667 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-6h6nq" podStartSLOduration=1.628442478 podStartE2EDuration="20.805650075s" totalImagesPullingTime="19.177207597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:58.804585748 +0000 UTC m=+168.239570373" watchObservedRunningTime="2026-07-14 14:32:58.805650075 +0000 UTC m=+168.240634723" Jul 14 14:32:58.827274 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:32:58.826336 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-794fbf756f-2s5wn" podStartSLOduration=3.826313925 podStartE2EDuration="3.826313925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:58.825956681 +0000 UTC m=+168.260941330" watchObservedRunningTime="2026-07-14 14:32:58.826313925 +0000 UTC m=+168.261298550" Jul 14 14:33:01.785474 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:01.785432 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzwj6" event={"ID":"bc7a7851-c2e6-414a-9716-5296d9c8e416","Type":"ContainerStarted","Data":"640e2145fae26caa47b1a31e232103be872a6c6a997b9e0c2aa26097605ece25"} Jul 14 14:33:01.785941 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:01.785503 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzwj6" event={"ID":"bc7a7851-c2e6-414a-9716-5296d9c8e416","Type":"ContainerStarted","Data":"61f5900f3ce3ef6ebfb594421ee5d2b55605cb22d4bdf0f17fa985994db57e1f"} Jul 14 14:33:01.785941 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:01.785541 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hzwj6" Jul 14 14:33:01.804759 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:01.804698 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hzwj6" podStartSLOduration=134.412818323 podStartE2EDuration="2m16.804679267s" totalImagesPullingTime="2.391860944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:33:01.802688474 +0000 UTC m=+171.237673100" watchObservedRunningTime="2026-07-14 14:33:01.804679267 +0000 UTC m=+171.239663890" Jul 14 14:33:05.467759 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:05.467653 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:33:05.467759 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:05.467737 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:33:05.473077 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:05.473050 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:33:05.806778 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:05.806741 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:33:05.855574 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:05.855540 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4b5cd577-dm8fm"] Jul 14 14:33:08.231352 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:08.231308 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:33:11.792544 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:11.792511 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hzwj6" Jul 14 14:33:30.887462 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:30.887418 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b4b5cd577-dm8fm" podUID="e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" containerName="console" containerID="cri-o://324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2" gracePeriod=15 Jul 14 14:33:31.161162 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.161080 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:33:31.321057 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321023 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-serving-cert\") pod \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " Jul 14 14:33:31.321057 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321063 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpxxt\" (UniqueName: \"kubernetes.io/projected/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-kube-api-access-qpxxt\") pod \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " Jul 14 14:33:31.321310 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321089 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-oauth-config\") pod \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " Jul 14 14:33:31.321310 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321145 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-oauth-serving-cert\") pod \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " Jul 14 14:33:31.321310 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321199 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-service-ca\") pod \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " Jul 14 14:33:31.321310 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321236 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-config\") pod \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\" (UID: \"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5\") " Jul 14 14:33:31.321653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321620 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" (UID: "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:33:31.321653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321637 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-service-ca" (OuterVolumeSpecName: "service-ca") pod "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" (UID: "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:33:31.321794 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.321657 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-config" (OuterVolumeSpecName: "console-config") pod "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" (UID: "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:33:31.323408 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.323379 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" (UID: "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:33:31.323497 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.323427 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-kube-api-access-qpxxt" (OuterVolumeSpecName: "kube-api-access-qpxxt") pod "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" (UID: "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5"). InnerVolumeSpecName "kube-api-access-qpxxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:33:31.323497 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.323480 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" (UID: "e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:33:31.422757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.422670 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-serving-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:33:31.422757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.422701 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpxxt\" (UniqueName: \"kubernetes.io/projected/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-kube-api-access-qpxxt\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:33:31.422757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.422711 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-oauth-config\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:33:31.422757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.422719 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-oauth-serving-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:33:31.422757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.422728 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-service-ca\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:33:31.422757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.422737 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5-console-config\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:33:31.895796 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.895758 2550 generic.go:356] "Generic (PLEG): container finished" podID="e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" containerID="324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2" exitCode=0 Jul 14 14:33:31.896294 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.895805 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b5cd577-dm8fm" Jul 14 14:33:31.896294 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.895842 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b5cd577-dm8fm" event={"ID":"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5","Type":"ContainerDied","Data":"324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2"} Jul 14 14:33:31.896294 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.895886 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b5cd577-dm8fm" event={"ID":"e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5","Type":"ContainerDied","Data":"e0d386b843bba06734cdd5c87d1a973188ffad88fff1629e747696b956d0ac07"} Jul 14 14:33:31.896294 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.895905 2550 scope.go:117] "RemoveContainer" containerID="324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2" Jul 14 14:33:31.915290 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.915271 2550 scope.go:117] "RemoveContainer" containerID="324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2" Jul 14 14:33:31.915675 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:33:31.915642 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2\": container with ID starting with 324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2 not found: ID does not exist" containerID="324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2" Jul 14 14:33:31.915812 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.915687 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2"} err="failed to get container status \"324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2\": rpc error: code = NotFound desc = could not find container \"324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2\": container with ID starting with 324e8c80d319ebe8b644581293691e3b6bdc96e3c0d435aed848897fa76880e2 not found: ID does not exist" Jul 14 14:33:31.917106 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.917080 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4b5cd577-dm8fm"] Jul 14 14:33:31.920516 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:31.920495 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b4b5cd577-dm8fm"] Jul 14 14:33:33.118665 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:33:33.118631 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" path="/var/lib/kubelet/pods/e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5/volumes" Jul 14 14:34:17.935884 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.935850 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdb746689-l2m7b"] Jul 14 14:34:17.936314 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.936104 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" containerName="console" Jul 14 14:34:17.936314 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.936114 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" containerName="console" Jul 14 14:34:17.936314 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.936166 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99a4c9d-6dce-4fa8-bfb0-cf8a30e8d3e5" containerName="console" Jul 14 14:34:17.938288 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.938272 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:17.950363 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.950320 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdb746689-l2m7b"] Jul 14 14:34:17.957427 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.957398 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-config\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:17.957565 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.957448 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-oauth-config\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:17.957565 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.957508 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-trusted-ca-bundle\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:17.957565 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.957555 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-serving-cert\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:17.957712 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.957580 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-service-ca\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:17.957712 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.957603 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-oauth-serving-cert\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:17.957712 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:17.957650 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmsg\" (UniqueName: \"kubernetes.io/projected/27419db7-a41c-48ce-83de-8dd3b60f9f1c-kube-api-access-gtmsg\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.058228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.058197 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-trusted-ca-bundle\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.058228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.058232 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-serving-cert\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.058495 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.058247 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-service-ca\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.058495 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.058263 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-oauth-serving-cert\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.058495 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.058302 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmsg\" (UniqueName: \"kubernetes.io/projected/27419db7-a41c-48ce-83de-8dd3b60f9f1c-kube-api-access-gtmsg\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.058495 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.058359 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-config\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.058495 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.058413 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-oauth-config\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.059167 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.059140 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-service-ca\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.059301 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.059140 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-oauth-serving-cert\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.059301 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.059269 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-config\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.059412 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.059341 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-trusted-ca-bundle\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.060814 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.060792 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-oauth-config\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.060999 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.060980 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-serving-cert\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.067478 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.067457 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmsg\" (UniqueName: \"kubernetes.io/projected/27419db7-a41c-48ce-83de-8dd3b60f9f1c-kube-api-access-gtmsg\") pod \"console-5fdb746689-l2m7b\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.247642 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.247542 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:18.379254 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:18.379222 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdb746689-l2m7b"] Jul 14 14:34:19.059410 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:19.059370 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdb746689-l2m7b" event={"ID":"27419db7-a41c-48ce-83de-8dd3b60f9f1c","Type":"ContainerStarted","Data":"878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95"} Jul 14 14:34:19.059812 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:19.059424 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdb746689-l2m7b" event={"ID":"27419db7-a41c-48ce-83de-8dd3b60f9f1c","Type":"ContainerStarted","Data":"990629dcb623802329314d1735cb5bd9427e89167c4963037223d96efd550d68"} Jul 14 14:34:19.079219 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:19.079169 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fdb746689-l2m7b" podStartSLOduration=2.079151287 podStartE2EDuration="2.079151287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:34:19.077426379 +0000 UTC m=+248.512411002" watchObservedRunningTime="2026-07-14 14:34:19.079151287 +0000 UTC m=+248.514135913" Jul 14 14:34:28.248461 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:28.248423 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:28.248953 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:28.248510 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:28.253533 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:28.253507 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:29.096649 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:29.096618 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:34:29.144585 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:29.144547 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794fbf756f-2s5wn"] Jul 14 14:34:54.164767 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.164726 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-794fbf756f-2s5wn" podUID="58d3e5d4-4947-4731-b0ff-f82ed2b82494" containerName="console" containerID="cri-o://d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34" gracePeriod=15 Jul 14 14:34:54.405558 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.405529 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:34:54.528195 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528166 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-trusted-ca-bundle\") pod \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " Jul 14 14:34:54.528380 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528203 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnqg7\" (UniqueName: \"kubernetes.io/projected/58d3e5d4-4947-4731-b0ff-f82ed2b82494-kube-api-access-nnqg7\") pod \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " Jul 14 14:34:54.528380 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528234 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-oauth-config\") pod \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " Jul 14 14:34:54.528380 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528253 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-config\") pod \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " Jul 14 14:34:54.528380 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528307 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-serving-cert\") pod \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " Jul 14 14:34:54.528380 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528338 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-service-ca\") pod \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " Jul 14 14:34:54.528380 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528357 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-oauth-serving-cert\") pod \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\" (UID: \"58d3e5d4-4947-4731-b0ff-f82ed2b82494\") " Jul 14 14:34:54.528718 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528692 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "58d3e5d4-4947-4731-b0ff-f82ed2b82494" (UID: "58d3e5d4-4947-4731-b0ff-f82ed2b82494"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:34:54.528783 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528756 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-service-ca" (OuterVolumeSpecName: "service-ca") pod "58d3e5d4-4947-4731-b0ff-f82ed2b82494" (UID: "58d3e5d4-4947-4731-b0ff-f82ed2b82494"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:34:54.528847 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528752 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-config" (OuterVolumeSpecName: "console-config") pod "58d3e5d4-4947-4731-b0ff-f82ed2b82494" (UID: "58d3e5d4-4947-4731-b0ff-f82ed2b82494"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:34:54.528847 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.528812 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "58d3e5d4-4947-4731-b0ff-f82ed2b82494" (UID: "58d3e5d4-4947-4731-b0ff-f82ed2b82494"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:34:54.530654 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.530628 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "58d3e5d4-4947-4731-b0ff-f82ed2b82494" (UID: "58d3e5d4-4947-4731-b0ff-f82ed2b82494"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:34:54.530909 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.530885 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "58d3e5d4-4947-4731-b0ff-f82ed2b82494" (UID: "58d3e5d4-4947-4731-b0ff-f82ed2b82494"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:34:54.530952 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.530921 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d3e5d4-4947-4731-b0ff-f82ed2b82494-kube-api-access-nnqg7" (OuterVolumeSpecName: "kube-api-access-nnqg7") pod "58d3e5d4-4947-4731-b0ff-f82ed2b82494" (UID: "58d3e5d4-4947-4731-b0ff-f82ed2b82494"). InnerVolumeSpecName "kube-api-access-nnqg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:34:54.628894 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.628850 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-serving-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:34:54.628894 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.628882 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-service-ca\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:34:54.628894 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.628891 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-oauth-serving-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:34:54.628894 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.628901 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-trusted-ca-bundle\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:34:54.629213 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.628911 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnqg7\" (UniqueName: \"kubernetes.io/projected/58d3e5d4-4947-4731-b0ff-f82ed2b82494-kube-api-access-nnqg7\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:34:54.629213 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.628919 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-oauth-config\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:34:54.629213 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:54.628927 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58d3e5d4-4947-4731-b0ff-f82ed2b82494-console-config\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:34:55.183258 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.183225 2550 generic.go:356] "Generic (PLEG): container finished" podID="58d3e5d4-4947-4731-b0ff-f82ed2b82494" containerID="d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34" exitCode=0 Jul 14 14:34:55.183640 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.183276 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794fbf756f-2s5wn" Jul 14 14:34:55.183640 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.183318 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794fbf756f-2s5wn" event={"ID":"58d3e5d4-4947-4731-b0ff-f82ed2b82494","Type":"ContainerDied","Data":"d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34"} Jul 14 14:34:55.183640 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.183374 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794fbf756f-2s5wn" event={"ID":"58d3e5d4-4947-4731-b0ff-f82ed2b82494","Type":"ContainerDied","Data":"3d5ca4dfe1c2e949326db5edca1f606f606f28420c8cde5ab0f404546213d841"} Jul 14 14:34:55.183640 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.183400 2550 scope.go:117] "RemoveContainer" containerID="d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34" Jul 14 14:34:55.191764 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.191745 2550 scope.go:117] "RemoveContainer" containerID="d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34" Jul 14 14:34:55.192065 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:34:55.192045 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34\": container with ID starting with d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34 not found: ID does not exist" containerID="d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34" Jul 14 14:34:55.192134 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.192074 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34"} err="failed to get container status \"d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34\": rpc error: code = NotFound desc = could not find container \"d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34\": container with ID starting with d193f1f436f527260d798c06567f8f09ca7d38193facfeb831d8ec9dad2d4b34 not found: ID does not exist" Jul 14 14:34:55.200707 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.200677 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794fbf756f-2s5wn"] Jul 14 14:34:55.207216 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:55.207186 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-794fbf756f-2s5wn"] Jul 14 14:34:57.118643 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:34:57.118609 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d3e5d4-4947-4731-b0ff-f82ed2b82494" path="/var/lib/kubelet/pods/58d3e5d4-4947-4731-b0ff-f82ed2b82494/volumes" Jul 14 14:35:08.379656 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.379619 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w2gst"] Jul 14 14:35:08.380137 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.379871 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d3e5d4-4947-4731-b0ff-f82ed2b82494" containerName="console" Jul 14 14:35:08.380137 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.379883 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d3e5d4-4947-4731-b0ff-f82ed2b82494" containerName="console" Jul 14 14:35:08.380137 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.379940 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d3e5d4-4947-4731-b0ff-f82ed2b82494" containerName="console" Jul 14 14:35:08.381685 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.381667 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w2gst" Jul 14 14:35:08.385072 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.385049 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 14:35:08.386408 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.386389 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 14:35:08.386503 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.386425 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fvnqx\"" Jul 14 14:35:08.394573 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.394548 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w2gst"] Jul 14 14:35:08.535706 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.535670 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5cc54982-b4c6-4671-93fe-00f0a81d2c98-bound-sa-token\") pod \"cert-manager-55b4d954d5-w2gst\" (UID: \"5cc54982-b4c6-4671-93fe-00f0a81d2c98\") " pod="cert-manager/cert-manager-55b4d954d5-w2gst" Jul 14 14:35:08.535901 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.535760 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rqz\" (UniqueName: \"kubernetes.io/projected/5cc54982-b4c6-4671-93fe-00f0a81d2c98-kube-api-access-w5rqz\") pod \"cert-manager-55b4d954d5-w2gst\" (UID: \"5cc54982-b4c6-4671-93fe-00f0a81d2c98\") " pod="cert-manager/cert-manager-55b4d954d5-w2gst" Jul 14 14:35:08.637033 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.636935 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rqz\" (UniqueName: \"kubernetes.io/projected/5cc54982-b4c6-4671-93fe-00f0a81d2c98-kube-api-access-w5rqz\") pod \"cert-manager-55b4d954d5-w2gst\" (UID: \"5cc54982-b4c6-4671-93fe-00f0a81d2c98\") " pod="cert-manager/cert-manager-55b4d954d5-w2gst" Jul 14 14:35:08.637230 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.637035 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5cc54982-b4c6-4671-93fe-00f0a81d2c98-bound-sa-token\") pod \"cert-manager-55b4d954d5-w2gst\" (UID: \"5cc54982-b4c6-4671-93fe-00f0a81d2c98\") " pod="cert-manager/cert-manager-55b4d954d5-w2gst" Jul 14 14:35:08.645586 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.645560 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5cc54982-b4c6-4671-93fe-00f0a81d2c98-bound-sa-token\") pod \"cert-manager-55b4d954d5-w2gst\" (UID: \"5cc54982-b4c6-4671-93fe-00f0a81d2c98\") " pod="cert-manager/cert-manager-55b4d954d5-w2gst" Jul 14 14:35:08.645856 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.645837 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rqz\" (UniqueName: \"kubernetes.io/projected/5cc54982-b4c6-4671-93fe-00f0a81d2c98-kube-api-access-w5rqz\") pod \"cert-manager-55b4d954d5-w2gst\" (UID: \"5cc54982-b4c6-4671-93fe-00f0a81d2c98\") " pod="cert-manager/cert-manager-55b4d954d5-w2gst" Jul 14 14:35:08.689840 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.689788 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w2gst" Jul 14 14:35:08.815462 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:08.815430 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w2gst"] Jul 14 14:35:08.819581 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:35:08.819548 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cc54982_b4c6_4671_93fe_00f0a81d2c98.slice/crio-a763505aba54bfda9f010b1c2b3e74cc6c48cf912ba5c338e99ea9a4fd9359c7 WatchSource:0}: Error finding container a763505aba54bfda9f010b1c2b3e74cc6c48cf912ba5c338e99ea9a4fd9359c7: Status 404 returned error can't find the container with id a763505aba54bfda9f010b1c2b3e74cc6c48cf912ba5c338e99ea9a4fd9359c7 Jul 14 14:35:09.239733 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:09.239698 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w2gst" event={"ID":"5cc54982-b4c6-4671-93fe-00f0a81d2c98","Type":"ContainerStarted","Data":"a763505aba54bfda9f010b1c2b3e74cc6c48cf912ba5c338e99ea9a4fd9359c7"} Jul 14 14:35:10.166031 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.165994 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw"] Jul 14 14:35:10.172530 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.172501 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" Jul 14 14:35:10.175351 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.175325 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 14:35:10.175531 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.175507 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9lft8\"" Jul 14 14:35:10.176665 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.176525 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 14:35:10.178644 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.178619 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw"] Jul 14 14:35:10.249627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.249581 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxft\" (UniqueName: \"kubernetes.io/projected/2bb166fd-1d62-4ee7-b587-0343283cde3a-kube-api-access-vcxft\") pod \"openshift-lws-operator-fd8ccff4c-rsplw\" (UID: \"2bb166fd-1d62-4ee7-b587-0343283cde3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" Jul 14 14:35:10.249849 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.249695 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bb166fd-1d62-4ee7-b587-0343283cde3a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rsplw\" (UID: \"2bb166fd-1d62-4ee7-b587-0343283cde3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" Jul 14 14:35:10.350799 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.350738 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bb166fd-1d62-4ee7-b587-0343283cde3a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rsplw\" (UID: \"2bb166fd-1d62-4ee7-b587-0343283cde3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" Jul 14 14:35:10.350991 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.350824 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxft\" (UniqueName: \"kubernetes.io/projected/2bb166fd-1d62-4ee7-b587-0343283cde3a-kube-api-access-vcxft\") pod \"openshift-lws-operator-fd8ccff4c-rsplw\" (UID: \"2bb166fd-1d62-4ee7-b587-0343283cde3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" Jul 14 14:35:10.351183 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.351154 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bb166fd-1d62-4ee7-b587-0343283cde3a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rsplw\" (UID: \"2bb166fd-1d62-4ee7-b587-0343283cde3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" Jul 14 14:35:10.361480 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.361452 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxft\" (UniqueName: \"kubernetes.io/projected/2bb166fd-1d62-4ee7-b587-0343283cde3a-kube-api-access-vcxft\") pod \"openshift-lws-operator-fd8ccff4c-rsplw\" (UID: \"2bb166fd-1d62-4ee7-b587-0343283cde3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" Jul 14 14:35:10.485473 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:10.485398 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" Jul 14 14:35:11.585983 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:11.585937 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:35:11.586861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:11.586842 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:35:11.589568 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:11.589550 2550 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 14:35:11.700402 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:11.700341 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw"] Jul 14 14:35:11.705508 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:35:11.705477 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bb166fd_1d62_4ee7_b587_0343283cde3a.slice/crio-584e96e8e6f26ac76d1dc76b0e98a5944412a9583c96d69f5e137adf0b63b0ad WatchSource:0}: Error finding container 584e96e8e6f26ac76d1dc76b0e98a5944412a9583c96d69f5e137adf0b63b0ad: Status 404 returned error can't find the container with id 584e96e8e6f26ac76d1dc76b0e98a5944412a9583c96d69f5e137adf0b63b0ad Jul 14 14:35:12.253329 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:12.253292 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" event={"ID":"2bb166fd-1d62-4ee7-b587-0343283cde3a","Type":"ContainerStarted","Data":"584e96e8e6f26ac76d1dc76b0e98a5944412a9583c96d69f5e137adf0b63b0ad"} Jul 14 14:35:12.254627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:12.254601 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w2gst" event={"ID":"5cc54982-b4c6-4671-93fe-00f0a81d2c98","Type":"ContainerStarted","Data":"a7fc53ef694aca531097e245c3bea95d25f1db02387276810c6d915e024a7684"} Jul 14 14:35:12.271134 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:12.271075 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-w2gst" podStartSLOduration=1.490058233 podStartE2EDuration="4.271054448s" totalImagesPullingTime="2.780996215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:35:12.270800833 +0000 UTC m=+301.705785457" watchObservedRunningTime="2026-07-14 14:35:12.271054448 +0000 UTC m=+301.706039072" Jul 14 14:35:15.273684 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:15.273634 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" event={"ID":"2bb166fd-1d62-4ee7-b587-0343283cde3a","Type":"ContainerStarted","Data":"58206d34364d3f89aa2cd97199a88db0dcce33028a275a6686a735508a5d6141"} Jul 14 14:35:15.292942 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:15.292887 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rsplw" podStartSLOduration=2.787075238 podStartE2EDuration="5.292873043s" totalImagesPullingTime="2.505797805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:35:15.290656801 +0000 UTC m=+304.725641436" watchObservedRunningTime="2026-07-14 14:35:15.292873043 +0000 UTC m=+304.727857664" Jul 14 14:35:33.800312 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.800277 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst"] Jul 14 14:35:33.803165 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.803149 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.806757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.806737 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-62jlr\"" Jul 14 14:35:33.806867 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.806822 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 14:35:33.807136 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.807117 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 14:35:33.807176 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.807134 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 14:35:33.807226 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.807212 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 14:35:33.826508 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.826474 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3caef96a-f494-4379-bd7e-0d46ec936597-manifests\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.826671 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.826519 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caef96a-f494-4379-bd7e-0d46ec936597-tmp\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.826671 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.826548 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3caef96a-f494-4379-bd7e-0d46ec936597-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.826671 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.826578 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3caef96a-f494-4379-bd7e-0d46ec936597-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.826671 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.826622 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq476\" (UniqueName: \"kubernetes.io/projected/3caef96a-f494-4379-bd7e-0d46ec936597-kube-api-access-wq476\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.827449 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.827426 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst"] Jul 14 14:35:33.927368 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.927332 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3caef96a-f494-4379-bd7e-0d46ec936597-manifests\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.927368 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.927364 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caef96a-f494-4379-bd7e-0d46ec936597-tmp\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.927618 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.927383 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3caef96a-f494-4379-bd7e-0d46ec936597-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.927618 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.927402 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3caef96a-f494-4379-bd7e-0d46ec936597-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.927618 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.927438 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq476\" (UniqueName: \"kubernetes.io/projected/3caef96a-f494-4379-bd7e-0d46ec936597-kube-api-access-wq476\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.927801 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.927777 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3caef96a-f494-4379-bd7e-0d46ec936597-manifests\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.929719 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.929697 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caef96a-f494-4379-bd7e-0d46ec936597-tmp\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.929943 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.929924 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3caef96a-f494-4379-bd7e-0d46ec936597-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.930127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.930108 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3caef96a-f494-4379-bd7e-0d46ec936597-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:33.941538 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:33.941510 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq476\" (UniqueName: \"kubernetes.io/projected/3caef96a-f494-4379-bd7e-0d46ec936597-kube-api-access-wq476\") pod \"opendatahub-operator-controller-manager-b4469fb8c-mwtst\" (UID: \"3caef96a-f494-4379-bd7e-0d46ec936597\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:34.113026 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:34.112904 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:34.252516 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:34.252475 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst"] Jul 14 14:35:34.256136 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:35:34.256094 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3caef96a_f494_4379_bd7e_0d46ec936597.slice/crio-4728835fffd0ce50f6958b66d65ce7ddfa25cbdd8c7b2eb06894c1b7cfdbe241 WatchSource:0}: Error finding container 4728835fffd0ce50f6958b66d65ce7ddfa25cbdd8c7b2eb06894c1b7cfdbe241: Status 404 returned error can't find the container with id 4728835fffd0ce50f6958b66d65ce7ddfa25cbdd8c7b2eb06894c1b7cfdbe241 Jul 14 14:35:34.258056 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:34.258036 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 14:35:34.341530 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:34.341496 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" event={"ID":"3caef96a-f494-4379-bd7e-0d46ec936597","Type":"ContainerStarted","Data":"4728835fffd0ce50f6958b66d65ce7ddfa25cbdd8c7b2eb06894c1b7cfdbe241"} Jul 14 14:35:38.359624 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:38.359589 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" event={"ID":"3caef96a-f494-4379-bd7e-0d46ec936597","Type":"ContainerStarted","Data":"3d059610bc8cc3ccff612dfc8a0508304d5b7ce87987c725d3163d25bba9d1ac"} Jul 14 14:35:40.369583 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:40.369549 2550 generic.go:356] "Generic (PLEG): container finished" podID="3caef96a-f494-4379-bd7e-0d46ec936597" containerID="3d059610bc8cc3ccff612dfc8a0508304d5b7ce87987c725d3163d25bba9d1ac" exitCode=0 Jul 14 14:35:40.370004 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:40.369624 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" event={"ID":"3caef96a-f494-4379-bd7e-0d46ec936597","Type":"ContainerDied","Data":"3d059610bc8cc3ccff612dfc8a0508304d5b7ce87987c725d3163d25bba9d1ac"} Jul 14 14:35:41.375741 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:41.375654 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" event={"ID":"3caef96a-f494-4379-bd7e-0d46ec936597","Type":"ContainerStarted","Data":"2f7550015db6c3f42bcdeaf2831459d94042bba394c3f78b23e10534e7537a3e"} Jul 14 14:35:41.376154 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:41.375871 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:41.399122 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:41.399072 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" podStartSLOduration=2.561022073 podStartE2EDuration="8.399056348s" totalImagesPullingTime="4.752695149s" totalInitContainerRuntime="1.085339126s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:35:41.398269042 +0000 UTC m=+330.833253666" watchObservedRunningTime="2026-07-14 14:35:41.399056348 +0000 UTC m=+330.834041039" Jul 14 14:35:52.383461 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:52.383429 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-mwtst" Jul 14 14:35:54.579752 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.579716 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f65956896-6hpnv"] Jul 14 14:35:54.587869 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.587848 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:54.591521 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.591496 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 14:35:54.591643 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.591580 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 14:35:54.592563 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.592544 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 14:35:54.592670 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.592575 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 14:35:54.592670 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.592549 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pswpq\"" Jul 14 14:35:54.604428 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.604402 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f65956896-6hpnv"] Jul 14 14:35:54.686865 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.686830 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4639828a-55fc-4f29-934f-fcf82a5e6b39-tls-certs\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:54.687049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.686869 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhbd\" (UniqueName: \"kubernetes.io/projected/4639828a-55fc-4f29-934f-fcf82a5e6b39-kube-api-access-zkhbd\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:54.687049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.686908 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4639828a-55fc-4f29-934f-fcf82a5e6b39-tmp\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:54.788177 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.788135 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4639828a-55fc-4f29-934f-fcf82a5e6b39-tmp\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:54.788375 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.788212 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4639828a-55fc-4f29-934f-fcf82a5e6b39-tls-certs\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:54.788375 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.788245 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhbd\" (UniqueName: \"kubernetes.io/projected/4639828a-55fc-4f29-934f-fcf82a5e6b39-kube-api-access-zkhbd\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:54.788489 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:35:54.788376 2550 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 14 14:35:54.788489 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:35:54.788458 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4639828a-55fc-4f29-934f-fcf82a5e6b39-tls-certs podName:4639828a-55fc-4f29-934f-fcf82a5e6b39 nodeName:}" failed. No retries permitted until 2026-07-14 14:35:55.288435495 +0000 UTC m=+344.723420111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4639828a-55fc-4f29-934f-fcf82a5e6b39-tls-certs") pod "kube-auth-proxy-6f65956896-6hpnv" (UID: "4639828a-55fc-4f29-934f-fcf82a5e6b39") : secret "kube-auth-proxy-tls" not found Jul 14 14:35:54.790440 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.790416 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4639828a-55fc-4f29-934f-fcf82a5e6b39-tmp\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:54.806729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:54.806703 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhbd\" (UniqueName: \"kubernetes.io/projected/4639828a-55fc-4f29-934f-fcf82a5e6b39-kube-api-access-zkhbd\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:55.125624 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.125590 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8k4j"] Jul 14 14:35:55.128849 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.128832 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:55.131625 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.131598 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dpfz4\"" Jul 14 14:35:55.132058 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.132030 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 14 14:35:55.147444 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.147416 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8k4j"] Jul 14 14:35:55.191493 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.191454 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9d2\" (UniqueName: \"kubernetes.io/projected/19d99917-a23a-46ed-8510-33748cdba09d-kube-api-access-zn9d2\") pod \"odh-model-controller-7998c67dc4-z8k4j\" (UID: \"19d99917-a23a-46ed-8510-33748cdba09d\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:55.191644 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.191553 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d99917-a23a-46ed-8510-33748cdba09d-cert\") pod \"odh-model-controller-7998c67dc4-z8k4j\" (UID: \"19d99917-a23a-46ed-8510-33748cdba09d\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:55.292613 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.292573 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d99917-a23a-46ed-8510-33748cdba09d-cert\") pod \"odh-model-controller-7998c67dc4-z8k4j\" (UID: \"19d99917-a23a-46ed-8510-33748cdba09d\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:55.292835 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.292652 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4639828a-55fc-4f29-934f-fcf82a5e6b39-tls-certs\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:55.292835 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.292688 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9d2\" (UniqueName: \"kubernetes.io/projected/19d99917-a23a-46ed-8510-33748cdba09d-kube-api-access-zn9d2\") pod \"odh-model-controller-7998c67dc4-z8k4j\" (UID: \"19d99917-a23a-46ed-8510-33748cdba09d\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:55.292835 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:35:55.292744 2550 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 14 14:35:55.292835 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:35:55.292820 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d99917-a23a-46ed-8510-33748cdba09d-cert podName:19d99917-a23a-46ed-8510-33748cdba09d nodeName:}" failed. No retries permitted until 2026-07-14 14:35:55.792799809 +0000 UTC m=+345.227784414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/19d99917-a23a-46ed-8510-33748cdba09d-cert") pod "odh-model-controller-7998c67dc4-z8k4j" (UID: "19d99917-a23a-46ed-8510-33748cdba09d") : secret "odh-model-controller-webhook-cert" not found Jul 14 14:35:55.295407 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.295379 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4639828a-55fc-4f29-934f-fcf82a5e6b39-tls-certs\") pod \"kube-auth-proxy-6f65956896-6hpnv\" (UID: \"4639828a-55fc-4f29-934f-fcf82a5e6b39\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:55.313759 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.313725 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9d2\" (UniqueName: \"kubernetes.io/projected/19d99917-a23a-46ed-8510-33748cdba09d-kube-api-access-zn9d2\") pod \"odh-model-controller-7998c67dc4-z8k4j\" (UID: \"19d99917-a23a-46ed-8510-33748cdba09d\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:55.497381 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.497298 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" Jul 14 14:35:55.675980 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.675940 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f65956896-6hpnv"] Jul 14 14:35:55.678082 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:35:55.678059 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4639828a_55fc_4f29_934f_fcf82a5e6b39.slice/crio-b735b6dc6c4ebd8d11b9da20bbb6bcc80fc6b73b003a148e4bfb2d9171a2017e WatchSource:0}: Error finding container b735b6dc6c4ebd8d11b9da20bbb6bcc80fc6b73b003a148e4bfb2d9171a2017e: Status 404 returned error can't find the container with id b735b6dc6c4ebd8d11b9da20bbb6bcc80fc6b73b003a148e4bfb2d9171a2017e Jul 14 14:35:55.797694 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.797653 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d99917-a23a-46ed-8510-33748cdba09d-cert\") pod \"odh-model-controller-7998c67dc4-z8k4j\" (UID: \"19d99917-a23a-46ed-8510-33748cdba09d\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:55.800051 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:55.800033 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d99917-a23a-46ed-8510-33748cdba09d-cert\") pod \"odh-model-controller-7998c67dc4-z8k4j\" (UID: \"19d99917-a23a-46ed-8510-33748cdba09d\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:56.039215 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.039171 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:35:56.112268 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.112239 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w"] Jul 14 14:35:56.115994 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.115972 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.119099 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.118833 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 14:35:56.119099 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.118853 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 14:35:56.119099 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.118922 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 14:35:56.119363 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.119277 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 14:35:56.119363 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.119285 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h5r2l\"" Jul 14 14:35:56.132659 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.132632 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w"] Jul 14 14:35:56.199645 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.199612 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8k4j"] Jul 14 14:35:56.201934 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.201912 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-cacerts\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.202029 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.201984 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fcf51f-5946-4158-86cc-e3240274e8ff-local-certs\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.202085 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.202065 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.202135 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.202117 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snn7z\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-kube-api-access-snn7z\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.202186 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.202172 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-token\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.202235 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.202208 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.202312 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.202286 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.203232 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:35:56.203207 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d99917_a23a_46ed_8510_33748cdba09d.slice/crio-d3c6587967d6dfd048e171a26d9ab9e5c4807ec7c2bbf3df8914974394474a39 WatchSource:0}: Error finding container d3c6587967d6dfd048e171a26d9ab9e5c4807ec7c2bbf3df8914974394474a39: Status 404 returned error can't find the container with id d3c6587967d6dfd048e171a26d9ab9e5c4807ec7c2bbf3df8914974394474a39 Jul 14 14:35:56.303438 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.303339 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-token\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.303438 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.303386 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.303639 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.303443 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.303639 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.303517 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-cacerts\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.303750 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.303677 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fcf51f-5946-4158-86cc-e3240274e8ff-local-certs\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.303750 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.303724 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.303841 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.303764 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snn7z\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-kube-api-access-snn7z\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.304256 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.304206 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.305883 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.305856 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.305997 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.305913 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fcf51f-5946-4158-86cc-e3240274e8ff-local-certs\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.306366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.306346 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.306409 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.306354 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-cacerts\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.313606 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.313579 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-token\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.313814 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.313793 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snn7z\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-kube-api-access-snn7z\") pod \"istiod-openshift-gateway-7574459656-jgb4w\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.426748 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.426684 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:35:56.437025 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.436991 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" event={"ID":"19d99917-a23a-46ed-8510-33748cdba09d","Type":"ContainerStarted","Data":"d3c6587967d6dfd048e171a26d9ab9e5c4807ec7c2bbf3df8914974394474a39"} Jul 14 14:35:56.438226 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.438197 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" event={"ID":"4639828a-55fc-4f29-934f-fcf82a5e6b39","Type":"ContainerStarted","Data":"b735b6dc6c4ebd8d11b9da20bbb6bcc80fc6b73b003a148e4bfb2d9171a2017e"} Jul 14 14:35:56.562901 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:56.562807 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w"] Jul 14 14:35:56.567094 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:35:56.567068 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fcf51f_5946_4158_86cc_e3240274e8ff.slice/crio-4f18910749d892accb0020d2cc66e071c6c9787034d03b3a34c92b65c54f621d WatchSource:0}: Error finding container 4f18910749d892accb0020d2cc66e071c6c9787034d03b3a34c92b65c54f621d: Status 404 returned error can't find the container with id 4f18910749d892accb0020d2cc66e071c6c9787034d03b3a34c92b65c54f621d Jul 14 14:35:57.445341 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:35:57.445297 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" event={"ID":"17fcf51f-5946-4158-86cc-e3240274e8ff","Type":"ContainerStarted","Data":"4f18910749d892accb0020d2cc66e071c6c9787034d03b3a34c92b65c54f621d"} Jul 14 14:36:01.223789 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.223753 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:36:01.224066 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.223824 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:36:01.463983 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.463925 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" event={"ID":"4639828a-55fc-4f29-934f-fcf82a5e6b39","Type":"ContainerStarted","Data":"0b23b27bab3d5619a631839ebfda107454d03fe919d4e7094200a7b8dc7cf2f5"} Jul 14 14:36:01.465528 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.465503 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" event={"ID":"19d99917-a23a-46ed-8510-33748cdba09d","Type":"ContainerStarted","Data":"a3a056f594047548bbd659d62a783dfbfe813370e11b77e9fa0cf81442fa7ad0"} Jul 14 14:36:01.465651 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.465554 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:36:01.466926 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.466907 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" event={"ID":"17fcf51f-5946-4158-86cc-e3240274e8ff","Type":"ContainerStarted","Data":"2445a699ee029c047255eeca65e03307cfb38e359d3f7cf7066285a922380d95"} Jul 14 14:36:01.467029 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.467018 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:36:01.490732 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.490643 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f65956896-6hpnv" podStartSLOduration=1.952881826 podStartE2EDuration="7.490628847s" totalImagesPullingTime="5.537747021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:36:01.488741933 +0000 UTC m=+350.923726559" watchObservedRunningTime="2026-07-14 14:36:01.490628847 +0000 UTC m=+350.925613485" Jul 14 14:36:01.508862 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.508804 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" podStartSLOduration=1.443280474 podStartE2EDuration="6.508782476s" totalImagesPullingTime="5.065502002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:36:01.507575892 +0000 UTC m=+350.942560517" watchObservedRunningTime="2026-07-14 14:36:01.508782476 +0000 UTC m=+350.943767105" Jul 14 14:36:01.534236 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:01.534186 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" podStartSLOduration=0.879709335 podStartE2EDuration="5.53416786s" totalImagesPullingTime="4.654458525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:36:01.532452624 +0000 UTC m=+350.967437251" watchObservedRunningTime="2026-07-14 14:36:01.53416786 +0000 UTC m=+350.969152484" Jul 14 14:36:02.473788 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:02.473752 2550 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-jgb4w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 14:36:02.474169 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:02.473859 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" podUID="17fcf51f-5946-4158-86cc-e3240274e8ff" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 14:36:03.273426 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.273321 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq"] Jul 14 14:36:03.277669 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.277646 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.283267 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.283235 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-m8qzn\"" Jul 14 14:36:03.300320 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.300288 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq"] Jul 14 14:36:03.368108 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368071 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413f0850-488d-484b-ba18-36d661a1aec8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.368301 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368116 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.368301 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368189 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.368301 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368233 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413f0850-488d-484b-ba18-36d661a1aec8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.368301 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368252 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmfjh\" (UniqueName: \"kubernetes.io/projected/413f0850-488d-484b-ba18-36d661a1aec8-kube-api-access-rmfjh\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.368502 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368341 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.368502 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368390 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.368502 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368412 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.368502 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.368432 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413f0850-488d-484b-ba18-36d661a1aec8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.469647 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469605 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413f0850-488d-484b-ba18-36d661a1aec8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.469647 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469651 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.469867 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469677 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.469867 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469704 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413f0850-488d-484b-ba18-36d661a1aec8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.469867 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469731 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmfjh\" (UniqueName: \"kubernetes.io/projected/413f0850-488d-484b-ba18-36d661a1aec8-kube-api-access-rmfjh\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.469867 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469774 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.469867 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469825 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.469867 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469848 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.470244 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.469876 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413f0850-488d-484b-ba18-36d661a1aec8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.470244 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.470201 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.470354 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.470312 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.470498 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.470465 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.470558 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.470536 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.470627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.470612 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413f0850-488d-484b-ba18-36d661a1aec8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.472256 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.472233 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413f0850-488d-484b-ba18-36d661a1aec8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.472618 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.472599 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413f0850-488d-484b-ba18-36d661a1aec8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.496231 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.496198 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmfjh\" (UniqueName: \"kubernetes.io/projected/413f0850-488d-484b-ba18-36d661a1aec8-kube-api-access-rmfjh\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.498165 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.498134 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413f0850-488d-484b-ba18-36d661a1aec8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447drmgjq\" (UID: \"413f0850-488d-484b-ba18-36d661a1aec8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.588935 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.588847 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:03.738160 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:03.738130 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq"] Jul 14 14:36:03.740744 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:36:03.740710 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413f0850_488d_484b_ba18_36d661a1aec8.slice/crio-9799820d75150e687e51bae76ffcfe105613b549e754b5f36bddcb5d6fca0c41 WatchSource:0}: Error finding container 9799820d75150e687e51bae76ffcfe105613b549e754b5f36bddcb5d6fca0c41: Status 404 returned error can't find the container with id 9799820d75150e687e51bae76ffcfe105613b549e754b5f36bddcb5d6fca0c41 Jul 14 14:36:04.481240 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:04.481204 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" event={"ID":"413f0850-488d-484b-ba18-36d661a1aec8","Type":"ContainerStarted","Data":"9799820d75150e687e51bae76ffcfe105613b549e754b5f36bddcb5d6fca0c41"} Jul 14 14:36:05.473372 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.473343 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:36:05.683881 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.683841 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-42jgk"] Jul 14 14:36:05.691281 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.691103 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:05.694034 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.694003 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 14 14:36:05.694366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.694339 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr"] Jul 14 14:36:05.694618 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.694601 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rw4rm\"" Jul 14 14:36:05.697866 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.697842 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:05.700580 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.700406 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vkrhm\"" Jul 14 14:36:05.701371 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.701352 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 14 14:36:05.716135 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.714563 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-42jgk"] Jul 14 14:36:05.716442 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.716417 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr"] Jul 14 14:36:05.791464 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.791412 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-g92jr\" (UID: \"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:05.791665 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.791482 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsk7q\" (UniqueName: \"kubernetes.io/projected/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-kube-api-access-jsk7q\") pod \"llmisvc-controller-manager-777c6bd7d5-g92jr\" (UID: \"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:05.791665 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.791533 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25fb7\" (UniqueName: \"kubernetes.io/projected/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-kube-api-access-25fb7\") pod \"kserve-controller-manager-d6f4b65b8-42jgk\" (UID: \"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:05.791665 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.791624 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-cert\") pod \"kserve-controller-manager-d6f4b65b8-42jgk\" (UID: \"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:05.893090 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.893051 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsk7q\" (UniqueName: \"kubernetes.io/projected/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-kube-api-access-jsk7q\") pod \"llmisvc-controller-manager-777c6bd7d5-g92jr\" (UID: \"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:05.893295 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.893113 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fb7\" (UniqueName: \"kubernetes.io/projected/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-kube-api-access-25fb7\") pod \"kserve-controller-manager-d6f4b65b8-42jgk\" (UID: \"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:05.893295 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.893171 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-cert\") pod \"kserve-controller-manager-d6f4b65b8-42jgk\" (UID: \"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:05.893295 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.893226 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-g92jr\" (UID: \"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:05.893457 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:36:05.893310 2550 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 14 14:36:05.893457 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:36:05.893329 2550 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 14 14:36:05.893457 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:36:05.893376 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-cert podName:b5382ae7-ea4a-44df-bb08-9f2f28e8fea5 nodeName:}" failed. No retries permitted until 2026-07-14 14:36:06.393355194 +0000 UTC m=+355.828339807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-cert") pod "kserve-controller-manager-d6f4b65b8-42jgk" (UID: "b5382ae7-ea4a-44df-bb08-9f2f28e8fea5") : secret "kserve-webhook-server-cert" not found Jul 14 14:36:05.893457 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:36:05.893397 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-cert podName:b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd nodeName:}" failed. No retries permitted until 2026-07-14 14:36:06.393387465 +0000 UTC m=+355.828372067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-cert") pod "llmisvc-controller-manager-777c6bd7d5-g92jr" (UID: "b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd") : secret "llmisvc-webhook-server-cert" not found Jul 14 14:36:05.916214 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.916177 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fb7\" (UniqueName: \"kubernetes.io/projected/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-kube-api-access-25fb7\") pod \"kserve-controller-manager-d6f4b65b8-42jgk\" (UID: \"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:05.916512 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:05.916493 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsk7q\" (UniqueName: \"kubernetes.io/projected/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-kube-api-access-jsk7q\") pod \"llmisvc-controller-manager-777c6bd7d5-g92jr\" (UID: \"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:06.398277 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.398237 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-cert\") pod \"kserve-controller-manager-d6f4b65b8-42jgk\" (UID: \"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:06.398461 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.398316 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-g92jr\" (UID: \"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:06.401352 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.401325 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-g92jr\" (UID: \"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:06.401463 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.401448 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5382ae7-ea4a-44df-bb08-9f2f28e8fea5-cert\") pod \"kserve-controller-manager-d6f4b65b8-42jgk\" (UID: \"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:06.549242 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.549207 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:36:06.549489 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.549276 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:36:06.549489 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.549302 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:36:06.607983 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.607937 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:06.616718 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.616685 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:06.787982 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.787929 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-42jgk"] Jul 14 14:36:06.792456 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:36:06.792425 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5382ae7_ea4a_44df_bb08_9f2f28e8fea5.slice/crio-1e71dfc60d6b3a62914f442e42e50f42d87757f585c3782f57dff4b26701c7d5 WatchSource:0}: Error finding container 1e71dfc60d6b3a62914f442e42e50f42d87757f585c3782f57dff4b26701c7d5: Status 404 returned error can't find the container with id 1e71dfc60d6b3a62914f442e42e50f42d87757f585c3782f57dff4b26701c7d5 Jul 14 14:36:06.797085 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:06.797063 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr"] Jul 14 14:36:06.799044 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:36:06.798986 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb70536a7_1cfa_4a3e_a46e_d1ad6c468cbd.slice/crio-463d154e6aae229fcbce35c4696ffd45483c2191874f615a16c16b1cfd7b27d9 WatchSource:0}: Error finding container 463d154e6aae229fcbce35c4696ffd45483c2191874f615a16c16b1cfd7b27d9: Status 404 returned error can't find the container with id 463d154e6aae229fcbce35c4696ffd45483c2191874f615a16c16b1cfd7b27d9 Jul 14 14:36:07.497505 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:07.497465 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" event={"ID":"413f0850-488d-484b-ba18-36d661a1aec8","Type":"ContainerStarted","Data":"7829953eaed6742c5af0547881ec0782ab6d4478fdb7e217f5a814e0cab38536"} Jul 14 14:36:07.498752 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:07.498712 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" event={"ID":"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5","Type":"ContainerStarted","Data":"1e71dfc60d6b3a62914f442e42e50f42d87757f585c3782f57dff4b26701c7d5"} Jul 14 14:36:07.499784 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:07.499764 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" event={"ID":"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd","Type":"ContainerStarted","Data":"463d154e6aae229fcbce35c4696ffd45483c2191874f615a16c16b1cfd7b27d9"} Jul 14 14:36:07.528075 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:07.527996 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" podStartSLOduration=1.721924429 podStartE2EDuration="4.527973039s" totalImagesPullingTime="2.80604861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:36:07.527791125 +0000 UTC m=+356.962775750" watchObservedRunningTime="2026-07-14 14:36:07.527973039 +0000 UTC m=+356.962957660" Jul 14 14:36:07.589595 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:07.589554 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:07.594743 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:07.594713 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:08.505983 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:08.505936 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:08.507134 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:08.507109 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447drmgjq" Jul 14 14:36:11.521882 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:11.521844 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" event={"ID":"b5382ae7-ea4a-44df-bb08-9f2f28e8fea5","Type":"ContainerStarted","Data":"82f71657e1cf1a53622da1ab430930cf5c0be432c2d3db471d597ce5b7650dfc"} Jul 14 14:36:11.522432 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:11.521907 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:11.523144 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:11.523117 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" event={"ID":"b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd","Type":"ContainerStarted","Data":"7ab062da0ca4ca6aea62d22d24a7bf3c07d730e6d68eaf34496df3619fa877d5"} Jul 14 14:36:11.523250 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:11.523167 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:11.539671 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:11.539615 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" podStartSLOduration=2.417042582 podStartE2EDuration="6.53960168s" totalImagesPullingTime="4.122559098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:36:11.539389546 +0000 UTC m=+360.974374169" watchObservedRunningTime="2026-07-14 14:36:11.53960168 +0000 UTC m=+360.974586304" Jul 14 14:36:11.557932 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:11.557869 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" podStartSLOduration=2.444376872 podStartE2EDuration="6.557848118s" totalImagesPullingTime="4.113471246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:36:11.556658032 +0000 UTC m=+360.991642655" watchObservedRunningTime="2026-07-14 14:36:11.557848118 +0000 UTC m=+360.992832742" Jul 14 14:36:12.474385 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:12.474347 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-z8k4j" Jul 14 14:36:18.438915 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.438879 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w"] Jul 14 14:36:18.439338 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.439117 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" podUID="17fcf51f-5946-4158-86cc-e3240274e8ff" containerName="discovery" containerID="cri-o://2445a699ee029c047255eeca65e03307cfb38e359d3f7cf7066285a922380d95" gracePeriod=30 Jul 14 14:36:18.561495 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.561469 2550 generic.go:356] "Generic (PLEG): container finished" podID="17fcf51f-5946-4158-86cc-e3240274e8ff" containerID="2445a699ee029c047255eeca65e03307cfb38e359d3f7cf7066285a922380d95" exitCode=0 Jul 14 14:36:18.561613 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.561544 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" event={"ID":"17fcf51f-5946-4158-86cc-e3240274e8ff","Type":"ContainerDied","Data":"2445a699ee029c047255eeca65e03307cfb38e359d3f7cf7066285a922380d95"} Jul 14 14:36:18.687222 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.687191 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:36:18.811581 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.811544 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-cacerts\") pod \"17fcf51f-5946-4158-86cc-e3240274e8ff\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " Jul 14 14:36:18.811776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.811595 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fcf51f-5946-4158-86cc-e3240274e8ff-local-certs\") pod \"17fcf51f-5946-4158-86cc-e3240274e8ff\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " Jul 14 14:36:18.811776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.811643 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-token\") pod \"17fcf51f-5946-4158-86cc-e3240274e8ff\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " Jul 14 14:36:18.811776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.811676 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-kubeconfig\") pod \"17fcf51f-5946-4158-86cc-e3240274e8ff\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " Jul 14 14:36:18.811776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.811740 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snn7z\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-kube-api-access-snn7z\") pod \"17fcf51f-5946-4158-86cc-e3240274e8ff\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " Jul 14 14:36:18.812041 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.811795 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-dns-cert\") pod \"17fcf51f-5946-4158-86cc-e3240274e8ff\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " Jul 14 14:36:18.812041 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.811822 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-ca-configmap\") pod \"17fcf51f-5946-4158-86cc-e3240274e8ff\" (UID: \"17fcf51f-5946-4158-86cc-e3240274e8ff\") " Jul 14 14:36:18.812385 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.812355 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "17fcf51f-5946-4158-86cc-e3240274e8ff" (UID: "17fcf51f-5946-4158-86cc-e3240274e8ff"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:36:18.814307 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.814278 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-cacerts" (OuterVolumeSpecName: "cacerts") pod "17fcf51f-5946-4158-86cc-e3240274e8ff" (UID: "17fcf51f-5946-4158-86cc-e3240274e8ff"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:36:18.814307 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.814291 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "17fcf51f-5946-4158-86cc-e3240274e8ff" (UID: "17fcf51f-5946-4158-86cc-e3240274e8ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:36:18.814457 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.814362 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "17fcf51f-5946-4158-86cc-e3240274e8ff" (UID: "17fcf51f-5946-4158-86cc-e3240274e8ff"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:36:18.814512 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.814452 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fcf51f-5946-4158-86cc-e3240274e8ff-local-certs" (OuterVolumeSpecName: "local-certs") pod "17fcf51f-5946-4158-86cc-e3240274e8ff" (UID: "17fcf51f-5946-4158-86cc-e3240274e8ff"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:36:18.814512 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.814466 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-kube-api-access-snn7z" (OuterVolumeSpecName: "kube-api-access-snn7z") pod "17fcf51f-5946-4158-86cc-e3240274e8ff" (UID: "17fcf51f-5946-4158-86cc-e3240274e8ff"). InnerVolumeSpecName "kube-api-access-snn7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:36:18.814612 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.814505 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "17fcf51f-5946-4158-86cc-e3240274e8ff" (UID: "17fcf51f-5946-4158-86cc-e3240274e8ff"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:36:18.913101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.913059 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snn7z\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-kube-api-access-snn7z\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:36:18.913101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.913090 2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-dns-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:36:18.913101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.913099 2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-csr-ca-configmap\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:36:18.913101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.913108 2550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-cacerts\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:36:18.913445 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.913117 2550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fcf51f-5946-4158-86cc-e3240274e8ff-local-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:36:18.913445 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.913148 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:36:18.913445 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:18.913156 2550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fcf51f-5946-4158-86cc-e3240274e8ff-istio-kubeconfig\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:36:19.566799 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:19.566767 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" Jul 14 14:36:19.566799 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:19.566792 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w" event={"ID":"17fcf51f-5946-4158-86cc-e3240274e8ff","Type":"ContainerDied","Data":"4f18910749d892accb0020d2cc66e071c6c9787034d03b3a34c92b65c54f621d"} Jul 14 14:36:19.567383 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:19.566844 2550 scope.go:117] "RemoveContainer" containerID="2445a699ee029c047255eeca65e03307cfb38e359d3f7cf7066285a922380d95" Jul 14 14:36:19.586904 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:19.586875 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w"] Jul 14 14:36:19.591001 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:19.590935 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jgb4w"] Jul 14 14:36:21.120547 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:21.120512 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fcf51f-5946-4158-86cc-e3240274e8ff" path="/var/lib/kubelet/pods/17fcf51f-5946-4158-86cc-e3240274e8ff/volumes" Jul 14 14:36:42.529116 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:42.529082 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g92jr" Jul 14 14:36:42.532075 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:42.532055 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-42jgk" Jul 14 14:36:55.715254 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.715221 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vgmnf"] Jul 14 14:36:55.715737 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.715511 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fcf51f-5946-4158-86cc-e3240274e8ff" containerName="discovery" Jul 14 14:36:55.715737 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.715522 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fcf51f-5946-4158-86cc-e3240274e8ff" containerName="discovery" Jul 14 14:36:55.715737 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.715584 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fcf51f-5946-4158-86cc-e3240274e8ff" containerName="discovery" Jul 14 14:36:55.722633 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.722605 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" Jul 14 14:36:55.725386 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.725363 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 14:36:55.726620 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.726602 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 14:36:55.726716 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.726608 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bs9hz\"" Jul 14 14:36:55.734708 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.734682 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vgmnf"] Jul 14 14:36:55.817727 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.817685 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr4zc\" (UniqueName: \"kubernetes.io/projected/4da027ac-1456-48a8-8e10-0bc17848162b-kube-api-access-fr4zc\") pod \"kuadrant-operator-catalog-vgmnf\" (UID: \"4da027ac-1456-48a8-8e10-0bc17848162b\") " pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" Jul 14 14:36:55.919045 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.918996 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr4zc\" (UniqueName: \"kubernetes.io/projected/4da027ac-1456-48a8-8e10-0bc17848162b-kube-api-access-fr4zc\") pod \"kuadrant-operator-catalog-vgmnf\" (UID: \"4da027ac-1456-48a8-8e10-0bc17848162b\") " pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" Jul 14 14:36:55.927538 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:55.927510 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr4zc\" (UniqueName: \"kubernetes.io/projected/4da027ac-1456-48a8-8e10-0bc17848162b-kube-api-access-fr4zc\") pod \"kuadrant-operator-catalog-vgmnf\" (UID: \"4da027ac-1456-48a8-8e10-0bc17848162b\") " pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" Jul 14 14:36:56.034073 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.034031 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" Jul 14 14:36:56.077202 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.077167 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vgmnf"] Jul 14 14:36:56.166255 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.166223 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vgmnf"] Jul 14 14:36:56.169102 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:36:56.169071 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da027ac_1456_48a8_8e10_0bc17848162b.slice/crio-b927a778d60134e27c336328e3b5361039d6f030f35091c99e25030180d5d8f9 WatchSource:0}: Error finding container b927a778d60134e27c336328e3b5361039d6f030f35091c99e25030180d5d8f9: Status 404 returned error can't find the container with id b927a778d60134e27c336328e3b5361039d6f030f35091c99e25030180d5d8f9 Jul 14 14:36:56.289402 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.289321 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xls6f"] Jul 14 14:36:56.292192 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.292174 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:36:56.301258 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.301232 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xls6f"] Jul 14 14:36:56.422791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.422748 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lsq\" (UniqueName: \"kubernetes.io/projected/c9d3bd27-5de6-4be6-a06c-648ecab60c27-kube-api-access-f8lsq\") pod \"kuadrant-operator-catalog-xls6f\" (UID: \"c9d3bd27-5de6-4be6-a06c-648ecab60c27\") " pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:36:56.523870 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.523829 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lsq\" (UniqueName: \"kubernetes.io/projected/c9d3bd27-5de6-4be6-a06c-648ecab60c27-kube-api-access-f8lsq\") pod \"kuadrant-operator-catalog-xls6f\" (UID: \"c9d3bd27-5de6-4be6-a06c-648ecab60c27\") " pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:36:56.532123 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.532091 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lsq\" (UniqueName: \"kubernetes.io/projected/c9d3bd27-5de6-4be6-a06c-648ecab60c27-kube-api-access-f8lsq\") pod \"kuadrant-operator-catalog-xls6f\" (UID: \"c9d3bd27-5de6-4be6-a06c-648ecab60c27\") " pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:36:56.602135 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.602047 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:36:56.729325 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.729251 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" event={"ID":"4da027ac-1456-48a8-8e10-0bc17848162b","Type":"ContainerStarted","Data":"b927a778d60134e27c336328e3b5361039d6f030f35091c99e25030180d5d8f9"} Jul 14 14:36:56.745479 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:56.745453 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xls6f"] Jul 14 14:36:56.747948 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:36:56.747921 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d3bd27_5de6_4be6_a06c_648ecab60c27.slice/crio-809b8be55a6cf525b75b79383e3ea7d3cddd1737e8c66e6385274f449e736041 WatchSource:0}: Error finding container 809b8be55a6cf525b75b79383e3ea7d3cddd1737e8c66e6385274f449e736041: Status 404 returned error can't find the container with id 809b8be55a6cf525b75b79383e3ea7d3cddd1737e8c66e6385274f449e736041 Jul 14 14:36:57.735028 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:57.734995 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" event={"ID":"c9d3bd27-5de6-4be6-a06c-648ecab60c27","Type":"ContainerStarted","Data":"809b8be55a6cf525b75b79383e3ea7d3cddd1737e8c66e6385274f449e736041"} Jul 14 14:36:58.741538 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:58.741503 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" event={"ID":"4da027ac-1456-48a8-8e10-0bc17848162b","Type":"ContainerStarted","Data":"b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47"} Jul 14 14:36:58.742032 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:58.741546 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" podUID="4da027ac-1456-48a8-8e10-0bc17848162b" containerName="registry-server" containerID="cri-o://b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47" gracePeriod=2 Jul 14 14:36:58.743220 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:58.743194 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" event={"ID":"c9d3bd27-5de6-4be6-a06c-648ecab60c27","Type":"ContainerStarted","Data":"4c9883a450667c34c76a8200aa1612eaf8c1f674623095634e3b1fcead0bf272"} Jul 14 14:36:58.757337 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:58.757284 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" podStartSLOduration=1.357196482 podStartE2EDuration="3.757270705s" totalImagesPullingTime="2.400074223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:36:58.755861436 +0000 UTC m=+408.190846060" watchObservedRunningTime="2026-07-14 14:36:58.757270705 +0000 UTC m=+408.192255338" Jul 14 14:36:58.771708 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:58.771656 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" podStartSLOduration=0.949689493 podStartE2EDuration="2.771642485s" totalImagesPullingTime="1.821952992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:36:58.77071602 +0000 UTC m=+408.205700646" watchObservedRunningTime="2026-07-14 14:36:58.771642485 +0000 UTC m=+408.206627108" Jul 14 14:36:58.991865 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:58.991837 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" Jul 14 14:36:59.049231 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.049200 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr4zc\" (UniqueName: \"kubernetes.io/projected/4da027ac-1456-48a8-8e10-0bc17848162b-kube-api-access-fr4zc\") pod \"4da027ac-1456-48a8-8e10-0bc17848162b\" (UID: \"4da027ac-1456-48a8-8e10-0bc17848162b\") " Jul 14 14:36:59.051377 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.051347 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da027ac-1456-48a8-8e10-0bc17848162b-kube-api-access-fr4zc" (OuterVolumeSpecName: "kube-api-access-fr4zc") pod "4da027ac-1456-48a8-8e10-0bc17848162b" (UID: "4da027ac-1456-48a8-8e10-0bc17848162b"). InnerVolumeSpecName "kube-api-access-fr4zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:36:59.150074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.150039 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr4zc\" (UniqueName: \"kubernetes.io/projected/4da027ac-1456-48a8-8e10-0bc17848162b-kube-api-access-fr4zc\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:36:59.749153 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.749106 2550 generic.go:356] "Generic (PLEG): container finished" podID="4da027ac-1456-48a8-8e10-0bc17848162b" containerID="b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47" exitCode=0 Jul 14 14:36:59.749556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.749186 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" Jul 14 14:36:59.749556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.749198 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" event={"ID":"4da027ac-1456-48a8-8e10-0bc17848162b","Type":"ContainerDied","Data":"b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47"} Jul 14 14:36:59.749556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.749259 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vgmnf" event={"ID":"4da027ac-1456-48a8-8e10-0bc17848162b","Type":"ContainerDied","Data":"b927a778d60134e27c336328e3b5361039d6f030f35091c99e25030180d5d8f9"} Jul 14 14:36:59.749556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.749280 2550 scope.go:117] "RemoveContainer" containerID="b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47" Jul 14 14:36:59.758317 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.758297 2550 scope.go:117] "RemoveContainer" containerID="b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47" Jul 14 14:36:59.758593 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:36:59.758573 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47\": container with ID starting with b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47 not found: ID does not exist" containerID="b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47" Jul 14 14:36:59.758654 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.758607 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47"} err="failed to get container status \"b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47\": rpc error: code = NotFound desc = could not find container \"b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47\": container with ID starting with b760e03b8371d16cd7f8ec1717be56198c5b9b37a4836a5b74d5a13b0adc0d47 not found: ID does not exist" Jul 14 14:36:59.764714 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.764689 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vgmnf"] Jul 14 14:36:59.770919 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:36:59.770896 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vgmnf"] Jul 14 14:37:01.119763 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:01.119718 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4da027ac-1456-48a8-8e10-0bc17848162b" path="/var/lib/kubelet/pods/4da027ac-1456-48a8-8e10-0bc17848162b/volumes" Jul 14 14:37:06.602995 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:06.602942 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:37:06.603420 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:06.603017 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:37:06.625504 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:06.625470 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:37:06.805372 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:06.805342 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xls6f" Jul 14 14:37:27.295976 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.295932 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r"] Jul 14 14:37:27.296471 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.296288 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4da027ac-1456-48a8-8e10-0bc17848162b" containerName="registry-server" Jul 14 14:37:27.296471 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.296301 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4da027ac-1456-48a8-8e10-0bc17848162b" containerName="registry-server" Jul 14 14:37:27.296471 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.296354 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4da027ac-1456-48a8-8e10-0bc17848162b" containerName="registry-server" Jul 14 14:37:27.299948 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.299931 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" Jul 14 14:37:27.302834 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.302810 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 14 14:37:27.302992 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.302913 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5xtnq\"" Jul 14 14:37:27.308747 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.308725 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r"] Jul 14 14:37:27.491981 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.491921 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4zl\" (UniqueName: \"kubernetes.io/projected/768b3a77-486e-480c-94f7-0722aa4f35c5-kube-api-access-nh4zl\") pod \"dns-operator-controller-manager-864dd78c4c-s4f7r\" (UID: \"768b3a77-486e-480c-94f7-0722aa4f35c5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" Jul 14 14:37:27.593510 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.593416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4zl\" (UniqueName: \"kubernetes.io/projected/768b3a77-486e-480c-94f7-0722aa4f35c5-kube-api-access-nh4zl\") pod \"dns-operator-controller-manager-864dd78c4c-s4f7r\" (UID: \"768b3a77-486e-480c-94f7-0722aa4f35c5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" Jul 14 14:37:27.607312 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.607287 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4zl\" (UniqueName: \"kubernetes.io/projected/768b3a77-486e-480c-94f7-0722aa4f35c5-kube-api-access-nh4zl\") pod \"dns-operator-controller-manager-864dd78c4c-s4f7r\" (UID: \"768b3a77-486e-480c-94f7-0722aa4f35c5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" Jul 14 14:37:27.610021 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.610001 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" Jul 14 14:37:27.739662 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.739637 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r"] Jul 14 14:37:27.742213 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:37:27.742187 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768b3a77_486e_480c_94f7_0722aa4f35c5.slice/crio-3e679c125e6949cfca372ee6bd5396541d52b6f394de48a5432528f44b82def4 WatchSource:0}: Error finding container 3e679c125e6949cfca372ee6bd5396541d52b6f394de48a5432528f44b82def4: Status 404 returned error can't find the container with id 3e679c125e6949cfca372ee6bd5396541d52b6f394de48a5432528f44b82def4 Jul 14 14:37:27.872564 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:27.872475 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" event={"ID":"768b3a77-486e-480c-94f7-0722aa4f35c5","Type":"ContainerStarted","Data":"3e679c125e6949cfca372ee6bd5396541d52b6f394de48a5432528f44b82def4"} Jul 14 14:37:30.889903 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:30.889865 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" event={"ID":"768b3a77-486e-480c-94f7-0722aa4f35c5","Type":"ContainerStarted","Data":"3869b5ed9f3b05b283df2cdcfc45ae91cced686bf5dbb46445c976c60f393a93"} Jul 14 14:37:30.890309 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:30.889988 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" Jul 14 14:37:30.907561 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:30.907511 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" podStartSLOduration=1.6888224379999999 podStartE2EDuration="3.90749675s" totalImagesPullingTime="2.218674312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:37:30.906247151 +0000 UTC m=+440.341231786" watchObservedRunningTime="2026-07-14 14:37:30.90749675 +0000 UTC m=+440.342481432" Jul 14 14:37:32.441583 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.441546 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw"] Jul 14 14:37:32.448546 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.448521 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:32.451184 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.451165 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fzsbb\"" Jul 14 14:37:32.458037 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.458012 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw"] Jul 14 14:37:32.548548 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.548509 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9bf4597-894e-41c9-baad-16d24c7308ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" (UID: \"f9bf4597-894e-41c9-baad-16d24c7308ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:32.548735 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.548583 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vssch\" (UniqueName: \"kubernetes.io/projected/f9bf4597-894e-41c9-baad-16d24c7308ef-kube-api-access-vssch\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" (UID: \"f9bf4597-894e-41c9-baad-16d24c7308ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:32.649722 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.649688 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vssch\" (UniqueName: \"kubernetes.io/projected/f9bf4597-894e-41c9-baad-16d24c7308ef-kube-api-access-vssch\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" (UID: \"f9bf4597-894e-41c9-baad-16d24c7308ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:32.649824 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.649747 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9bf4597-894e-41c9-baad-16d24c7308ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" (UID: \"f9bf4597-894e-41c9-baad-16d24c7308ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:32.650102 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.650086 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9bf4597-894e-41c9-baad-16d24c7308ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" (UID: \"f9bf4597-894e-41c9-baad-16d24c7308ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:32.659905 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.659877 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vssch\" (UniqueName: \"kubernetes.io/projected/f9bf4597-894e-41c9-baad-16d24c7308ef-kube-api-access-vssch\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" (UID: \"f9bf4597-894e-41c9-baad-16d24c7308ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:32.760024 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.759989 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:32.901238 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:32.901211 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw"] Jul 14 14:37:32.903668 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:37:32.903644 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bf4597_894e_41c9_baad_16d24c7308ef.slice/crio-489cdc1460aa05e4ffb479be5bf7d5089a28ab4cd4169cf7b691106a134712a6 WatchSource:0}: Error finding container 489cdc1460aa05e4ffb479be5bf7d5089a28ab4cd4169cf7b691106a134712a6: Status 404 returned error can't find the container with id 489cdc1460aa05e4ffb479be5bf7d5089a28ab4cd4169cf7b691106a134712a6 Jul 14 14:37:33.904575 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:33.904540 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" event={"ID":"f9bf4597-894e-41c9-baad-16d24c7308ef","Type":"ContainerStarted","Data":"489cdc1460aa05e4ffb479be5bf7d5089a28ab4cd4169cf7b691106a134712a6"} Jul 14 14:37:36.420198 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.417876 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc"] Jul 14 14:37:36.422552 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.422521 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" Jul 14 14:37:36.426342 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.426318 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7jrzh\"" Jul 14 14:37:36.434072 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.434044 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc"] Jul 14 14:37:36.477831 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.477790 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrdd4\" (UniqueName: \"kubernetes.io/projected/47964024-b4b4-4e67-bc2a-635288b6b7af-kube-api-access-zrdd4\") pod \"authorino-operator-75cc8d5cb7-4z7mc\" (UID: \"47964024-b4b4-4e67-bc2a-635288b6b7af\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" Jul 14 14:37:36.578453 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.578409 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrdd4\" (UniqueName: \"kubernetes.io/projected/47964024-b4b4-4e67-bc2a-635288b6b7af-kube-api-access-zrdd4\") pod \"authorino-operator-75cc8d5cb7-4z7mc\" (UID: \"47964024-b4b4-4e67-bc2a-635288b6b7af\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" Jul 14 14:37:36.587749 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.587719 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrdd4\" (UniqueName: \"kubernetes.io/projected/47964024-b4b4-4e67-bc2a-635288b6b7af-kube-api-access-zrdd4\") pod \"authorino-operator-75cc8d5cb7-4z7mc\" (UID: \"47964024-b4b4-4e67-bc2a-635288b6b7af\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" Jul 14 14:37:36.736108 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.736007 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" Jul 14 14:37:36.897406 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.897373 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc"] Jul 14 14:37:36.900356 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:37:36.900334 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47964024_b4b4_4e67_bc2a_635288b6b7af.slice/crio-f9fe13c41d9f86edd7d105cb40f6ad6117fe19ee089bb2bbaae2dc6de4ac9294 WatchSource:0}: Error finding container f9fe13c41d9f86edd7d105cb40f6ad6117fe19ee089bb2bbaae2dc6de4ac9294: Status 404 returned error can't find the container with id f9fe13c41d9f86edd7d105cb40f6ad6117fe19ee089bb2bbaae2dc6de4ac9294 Jul 14 14:37:36.920504 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:36.920473 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" event={"ID":"47964024-b4b4-4e67-bc2a-635288b6b7af","Type":"ContainerStarted","Data":"f9fe13c41d9f86edd7d105cb40f6ad6117fe19ee089bb2bbaae2dc6de4ac9294"} Jul 14 14:37:39.279890 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.279852 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t"] Jul 14 14:37:39.283777 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.283751 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:39.286766 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.286734 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jwj86\"" Jul 14 14:37:39.297394 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.297365 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t"] Jul 14 14:37:39.300710 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.300680 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wph6p\" (UniqueName: \"kubernetes.io/projected/242a6b0e-c77d-488e-a119-4e22b46baf3f-kube-api-access-wph6p\") pod \"limitador-operator-controller-manager-859d7b78bd-q479t\" (UID: \"242a6b0e-c77d-488e-a119-4e22b46baf3f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:39.402010 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.401954 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wph6p\" (UniqueName: \"kubernetes.io/projected/242a6b0e-c77d-488e-a119-4e22b46baf3f-kube-api-access-wph6p\") pod \"limitador-operator-controller-manager-859d7b78bd-q479t\" (UID: \"242a6b0e-c77d-488e-a119-4e22b46baf3f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:39.411287 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.411262 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wph6p\" (UniqueName: \"kubernetes.io/projected/242a6b0e-c77d-488e-a119-4e22b46baf3f-kube-api-access-wph6p\") pod \"limitador-operator-controller-manager-859d7b78bd-q479t\" (UID: \"242a6b0e-c77d-488e-a119-4e22b46baf3f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:39.596251 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.596169 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:39.727225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.727181 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t"] Jul 14 14:37:39.729453 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:37:39.729427 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242a6b0e_c77d_488e_a119_4e22b46baf3f.slice/crio-6636be0d9d0d8ea5503c7b4a49cee76cf38a0b341677bafca3eecc4868988763 WatchSource:0}: Error finding container 6636be0d9d0d8ea5503c7b4a49cee76cf38a0b341677bafca3eecc4868988763: Status 404 returned error can't find the container with id 6636be0d9d0d8ea5503c7b4a49cee76cf38a0b341677bafca3eecc4868988763 Jul 14 14:37:39.937144 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.937037 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" event={"ID":"242a6b0e-c77d-488e-a119-4e22b46baf3f","Type":"ContainerStarted","Data":"6636be0d9d0d8ea5503c7b4a49cee76cf38a0b341677bafca3eecc4868988763"} Jul 14 14:37:39.938516 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.938489 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" event={"ID":"47964024-b4b4-4e67-bc2a-635288b6b7af","Type":"ContainerStarted","Data":"e7dac326d18ed7c0d7354ae220a9a77cc3c718f53be36279d16fef882a7ee658"} Jul 14 14:37:39.938637 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.938605 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" Jul 14 14:37:39.959247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:39.959202 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" podStartSLOduration=1.492250758 podStartE2EDuration="3.959188113s" totalImagesPullingTime="2.466937355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:37:39.958305489 +0000 UTC m=+449.393290113" watchObservedRunningTime="2026-07-14 14:37:39.959188113 +0000 UTC m=+449.394172714" Jul 14 14:37:41.897329 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:41.897294 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s4f7r" Jul 14 14:37:42.954121 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:42.954085 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" event={"ID":"f9bf4597-894e-41c9-baad-16d24c7308ef","Type":"ContainerStarted","Data":"6df3f2d60bff56b032031a5ad6d416f5fec3e12ed22bed60e447fb444851d00f"} Jul 14 14:37:42.954479 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:42.954221 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:42.977019 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:42.976935 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" podStartSLOduration=1.9013603909999999 podStartE2EDuration="10.976919127s" totalImagesPullingTime="9.075558736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:37:42.975557414 +0000 UTC m=+452.410542041" watchObservedRunningTime="2026-07-14 14:37:42.976919127 +0000 UTC m=+452.411903755" Jul 14 14:37:43.960123 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:43.960085 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" event={"ID":"242a6b0e-c77d-488e-a119-4e22b46baf3f","Type":"ContainerStarted","Data":"7769518e8da3b0c9fd07f6679c8be57e56aea841c96493d2de513c2215da0a33"} Jul 14 14:37:43.960570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:43.960338 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:43.982266 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:43.982211 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" podStartSLOduration=1.7902434600000001 podStartE2EDuration="4.98219592s" totalImagesPullingTime="3.19195246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:37:43.980642044 +0000 UTC m=+453.415626669" watchObservedRunningTime="2026-07-14 14:37:43.98219592 +0000 UTC m=+453.417180641" Jul 14 14:37:44.914161 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.914116 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh"] Jul 14 14:37:44.917676 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.917648 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:44.920359 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.920327 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 14 14:37:44.920359 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.920355 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6c6jq\"" Jul 14 14:37:44.920569 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.920522 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 14 14:37:44.926744 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.926721 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh"] Jul 14 14:37:44.951205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.951161 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:44.951393 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.951224 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:44.951393 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:44.951360 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tp86\" (UniqueName: \"kubernetes.io/projected/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-kube-api-access-6tp86\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.052277 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.052234 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tp86\" (UniqueName: \"kubernetes.io/projected/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-kube-api-access-6tp86\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.052767 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.052302 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.052767 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.052339 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.052767 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:45.052500 2550 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 14 14:37:45.052767 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:45.052567 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-plugin-serving-cert podName:7e5222cd-dbf1-49ef-9a5d-fd9a347b609e nodeName:}" failed. No retries permitted until 2026-07-14 14:37:45.552548219 +0000 UTC m=+454.987532833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-s7rvh" (UID: "7e5222cd-dbf1-49ef-9a5d-fd9a347b609e") : secret "plugin-serving-cert" not found Jul 14 14:37:45.053106 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.053083 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.061746 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.061709 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tp86\" (UniqueName: \"kubernetes.io/projected/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-kube-api-access-6tp86\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.556985 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.556911 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.559415 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.559372 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e5222cd-dbf1-49ef-9a5d-fd9a347b609e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rvh\" (UID: \"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.829257 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.829141 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" Jul 14 14:37:45.961505 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.961482 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh"] Jul 14 14:37:45.963715 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:37:45.963683 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e5222cd_dbf1_49ef_9a5d_fd9a347b609e.slice/crio-35e289d4e0e5795e79df5bea120dc52c57c0b4681ca6833a45f431a3c665a648 WatchSource:0}: Error finding container 35e289d4e0e5795e79df5bea120dc52c57c0b4681ca6833a45f431a3c665a648: Status 404 returned error can't find the container with id 35e289d4e0e5795e79df5bea120dc52c57c0b4681ca6833a45f431a3c665a648 Jul 14 14:37:45.972654 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:45.972624 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" event={"ID":"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e","Type":"ContainerStarted","Data":"35e289d4e0e5795e79df5bea120dc52c57c0b4681ca6833a45f431a3c665a648"} Jul 14 14:37:50.945848 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:50.945814 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4z7mc" Jul 14 14:37:50.959782 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:50.959754 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d88fc5fb-xrzss"] Jul 14 14:37:50.975799 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:50.975751 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:50.976521 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:50.976491 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d88fc5fb-xrzss"] Jul 14 14:37:51.108167 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.108130 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrgv\" (UniqueName: \"kubernetes.io/projected/c16f4571-c50e-46dc-890f-91706ec57415-kube-api-access-vjrgv\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.108419 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.108240 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-trusted-ca-bundle\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.108419 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.108315 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-service-ca\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.108419 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.108358 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c16f4571-c50e-46dc-890f-91706ec57415-console-serving-cert\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.108419 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.108396 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-oauth-serving-cert\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.108632 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.108428 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-console-config\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.108632 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.108471 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c16f4571-c50e-46dc-890f-91706ec57415-console-oauth-config\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.209742 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.209436 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-trusted-ca-bundle\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.209742 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.209514 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-service-ca\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.209742 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.209556 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c16f4571-c50e-46dc-890f-91706ec57415-console-serving-cert\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.209742 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.209584 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-oauth-serving-cert\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.209742 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.209625 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-console-config\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.209742 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.209651 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c16f4571-c50e-46dc-890f-91706ec57415-console-oauth-config\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.209742 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.209691 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrgv\" (UniqueName: \"kubernetes.io/projected/c16f4571-c50e-46dc-890f-91706ec57415-kube-api-access-vjrgv\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.210837 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.210363 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-service-ca\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.210837 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.210414 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-trusted-ca-bundle\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.211064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.210848 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-oauth-serving-cert\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.211064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.210920 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c16f4571-c50e-46dc-890f-91706ec57415-console-config\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.212906 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.212867 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c16f4571-c50e-46dc-890f-91706ec57415-console-serving-cert\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.213314 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.213287 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c16f4571-c50e-46dc-890f-91706ec57415-console-oauth-config\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.224608 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.224577 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrgv\" (UniqueName: \"kubernetes.io/projected/c16f4571-c50e-46dc-890f-91706ec57415-kube-api-access-vjrgv\") pod \"console-74d88fc5fb-xrzss\" (UID: \"c16f4571-c50e-46dc-890f-91706ec57415\") " pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:51.288924 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:51.288872 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:37:52.388698 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:52.388634 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d88fc5fb-xrzss"] Jul 14 14:37:52.390841 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:37:52.390809 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16f4571_c50e_46dc_890f_91706ec57415.slice/crio-399901bce7d9b2f513c12ef00fd290f67748209376966cc4c9addd98a4224f5a WatchSource:0}: Error finding container 399901bce7d9b2f513c12ef00fd290f67748209376966cc4c9addd98a4224f5a: Status 404 returned error can't find the container with id 399901bce7d9b2f513c12ef00fd290f67748209376966cc4c9addd98a4224f5a Jul 14 14:37:53.011369 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:53.011333 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d88fc5fb-xrzss" event={"ID":"c16f4571-c50e-46dc-890f-91706ec57415","Type":"ContainerStarted","Data":"14e01f2310823460d78e921049ed15e76eb9ba4185a029309bf806119afe39ec"} Jul 14 14:37:53.011525 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:53.011383 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d88fc5fb-xrzss" event={"ID":"c16f4571-c50e-46dc-890f-91706ec57415","Type":"ContainerStarted","Data":"399901bce7d9b2f513c12ef00fd290f67748209376966cc4c9addd98a4224f5a"} Jul 14 14:37:53.031267 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:53.031215 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74d88fc5fb-xrzss" podStartSLOduration=3.03119795 podStartE2EDuration="3.03119795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:37:53.030067422 +0000 UTC m=+462.465052050" watchObservedRunningTime="2026-07-14 14:37:53.03119795 +0000 UTC m=+462.466182573" Jul 14 14:37:53.963133 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:53.963101 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:54.968185 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:54.968153 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:55.869475 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.869427 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw"] Jul 14 14:37:55.869674 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.869648 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" containerName="manager" containerID="cri-o://6df3f2d60bff56b032031a5ad6d416f5fec3e12ed22bed60e447fb444851d00f" gracePeriod=2 Jul 14 14:37:55.881814 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.881783 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw"] Jul 14 14:37:55.888335 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:55.888293 2550 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f9bf4597-894e-41c9-baad-16d24c7308ef\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\": pods \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:55.897037 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.897011 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t"] Jul 14 14:37:55.897281 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.897235 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" containerName="manager" containerID="cri-o://7769518e8da3b0c9fd07f6679c8be57e56aea841c96493d2de513c2215da0a33" gracePeriod=2 Jul 14 14:37:55.903659 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.903454 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv"] Jul 14 14:37:55.904343 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.904228 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" containerName="manager" Jul 14 14:37:55.904343 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.904253 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" containerName="manager" Jul 14 14:37:55.905040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.904543 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" containerName="manager" Jul 14 14:37:55.914398 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.914364 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:37:55.915924 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.915897 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t"] Jul 14 14:37:55.926566 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:55.926527 2550 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"242a6b0e-c77d-488e-a119-4e22b46baf3f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-q479t\": pods \"limitador-operator-controller-manager-859d7b78bd-q479t\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:55.929323 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:55.929286 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:55.932010 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.931934 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv"] Jul 14 14:37:55.933036 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:55.932919 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:55.934136 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.934112 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9"] Jul 14 14:37:55.934554 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.934537 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" containerName="manager" Jul 14 14:37:55.934554 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.934557 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" containerName="manager" Jul 14 14:37:55.934699 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.934657 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" containerName="manager" Jul 14 14:37:55.939542 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.939523 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" Jul 14 14:37:55.955146 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:55.955112 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9"] Jul 14 14:37:55.975632 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:55.975567 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q479t\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:55.978018 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:55.977978 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q479t\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:37:56.002493 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:37:56.002452 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:37:56.055211 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.055166 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xmvxv\" (UID: \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:37:56.055388 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.055253 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgwbp\" (UniqueName: \"kubernetes.io/projected/681b4a73-84d5-4c51-9c70-7608697f0ce4-kube-api-access-wgwbp\") pod \"limitador-operator-controller-manager-859d7b78bd-rq5l9\" (UID: \"681b4a73-84d5-4c51-9c70-7608697f0ce4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" Jul 14 14:37:56.055388 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.055296 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppnp\" (UniqueName: \"kubernetes.io/projected/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-kube-api-access-jppnp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xmvxv\" (UID: \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:37:56.156138 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.156053 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwbp\" (UniqueName: \"kubernetes.io/projected/681b4a73-84d5-4c51-9c70-7608697f0ce4-kube-api-access-wgwbp\") pod \"limitador-operator-controller-manager-859d7b78bd-rq5l9\" (UID: \"681b4a73-84d5-4c51-9c70-7608697f0ce4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" Jul 14 14:37:56.156138 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.156105 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppnp\" (UniqueName: \"kubernetes.io/projected/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-kube-api-access-jppnp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xmvxv\" (UID: \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:37:56.156361 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.156172 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xmvxv\" (UID: \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:37:56.156583 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.156548 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xmvxv\" (UID: \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:37:56.167197 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.167168 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwbp\" (UniqueName: \"kubernetes.io/projected/681b4a73-84d5-4c51-9c70-7608697f0ce4-kube-api-access-wgwbp\") pod \"limitador-operator-controller-manager-859d7b78bd-rq5l9\" (UID: \"681b4a73-84d5-4c51-9c70-7608697f0ce4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" Jul 14 14:37:56.167601 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.167582 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppnp\" (UniqueName: \"kubernetes.io/projected/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-kube-api-access-jppnp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xmvxv\" (UID: \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:37:56.290555 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.290514 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:37:56.297562 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:37:56.297527 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" Jul 14 14:38:01.289745 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:01.289704 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:38:01.290338 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:01.289772 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:38:01.295169 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:01.295143 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:38:02.064090 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:02.064058 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74d88fc5fb-xrzss" Jul 14 14:38:02.124314 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:02.124274 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdb746689-l2m7b"] Jul 14 14:38:10.107121 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.106997 2550 generic.go:356] "Generic (PLEG): container finished" podID="242a6b0e-c77d-488e-a119-4e22b46baf3f" containerID="7769518e8da3b0c9fd07f6679c8be57e56aea841c96493d2de513c2215da0a33" exitCode=0 Jul 14 14:38:10.109725 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.109627 2550 generic.go:356] "Generic (PLEG): container finished" podID="f9bf4597-894e-41c9-baad-16d24c7308ef" containerID="6df3f2d60bff56b032031a5ad6d416f5fec3e12ed22bed60e447fb444851d00f" exitCode=0 Jul 14 14:38:10.130433 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.130373 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv"] Jul 14 14:38:10.131446 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:38:10.131416 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b1e4b4_ad38_418c_96f5_a9ad6a68a6e8.slice/crio-358c8db73af31238263cec549a52db823de9dca1e10b96a1ebc19c963f271109 WatchSource:0}: Error finding container 358c8db73af31238263cec549a52db823de9dca1e10b96a1ebc19c963f271109: Status 404 returned error can't find the container with id 358c8db73af31238263cec549a52db823de9dca1e10b96a1ebc19c963f271109 Jul 14 14:38:10.132942 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.132915 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:38:10.135947 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:10.135916 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q479t\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:38:10.161821 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.161795 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9"] Jul 14 14:38:10.164158 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:38:10.164130 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681b4a73_84d5_4c51_9c70_7608697f0ce4.slice/crio-409d57bfb253b6116b9087ef296f022e9c06950cc5bba22f22e975c28d30f883 WatchSource:0}: Error finding container 409d57bfb253b6116b9087ef296f022e9c06950cc5bba22f22e975c28d30f883: Status 404 returned error can't find the container with id 409d57bfb253b6116b9087ef296f022e9c06950cc5bba22f22e975c28d30f883 Jul 14 14:38:10.173986 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.173953 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:38:10.176305 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:10.176280 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q479t\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:38:10.178407 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:10.178379 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:38:10.288067 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.288033 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9bf4597-894e-41c9-baad-16d24c7308ef-extensions-socket-volume\") pod \"f9bf4597-894e-41c9-baad-16d24c7308ef\" (UID: \"f9bf4597-894e-41c9-baad-16d24c7308ef\") " Jul 14 14:38:10.288250 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.288080 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vssch\" (UniqueName: \"kubernetes.io/projected/f9bf4597-894e-41c9-baad-16d24c7308ef-kube-api-access-vssch\") pod \"f9bf4597-894e-41c9-baad-16d24c7308ef\" (UID: \"f9bf4597-894e-41c9-baad-16d24c7308ef\") " Jul 14 14:38:10.288250 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.288160 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wph6p\" (UniqueName: \"kubernetes.io/projected/242a6b0e-c77d-488e-a119-4e22b46baf3f-kube-api-access-wph6p\") pod \"242a6b0e-c77d-488e-a119-4e22b46baf3f\" (UID: \"242a6b0e-c77d-488e-a119-4e22b46baf3f\") " Jul 14 14:38:10.288607 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.288576 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf4597-894e-41c9-baad-16d24c7308ef-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f9bf4597-894e-41c9-baad-16d24c7308ef" (UID: "f9bf4597-894e-41c9-baad-16d24c7308ef"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:38:10.290257 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.290217 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242a6b0e-c77d-488e-a119-4e22b46baf3f-kube-api-access-wph6p" (OuterVolumeSpecName: "kube-api-access-wph6p") pod "242a6b0e-c77d-488e-a119-4e22b46baf3f" (UID: "242a6b0e-c77d-488e-a119-4e22b46baf3f"). InnerVolumeSpecName "kube-api-access-wph6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:38:10.290590 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.290572 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bf4597-894e-41c9-baad-16d24c7308ef-kube-api-access-vssch" (OuterVolumeSpecName: "kube-api-access-vssch") pod "f9bf4597-894e-41c9-baad-16d24c7308ef" (UID: "f9bf4597-894e-41c9-baad-16d24c7308ef"). InnerVolumeSpecName "kube-api-access-vssch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:38:10.389285 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.389248 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9bf4597-894e-41c9-baad-16d24c7308ef-extensions-socket-volume\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:10.389285 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.389284 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vssch\" (UniqueName: \"kubernetes.io/projected/f9bf4597-894e-41c9-baad-16d24c7308ef-kube-api-access-vssch\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:10.389503 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:10.389298 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wph6p\" (UniqueName: \"kubernetes.io/projected/242a6b0e-c77d-488e-a119-4e22b46baf3f-kube-api-access-wph6p\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:11.118853 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:11.118820 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q479t\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:38:11.119947 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.119925 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:38:11.120534 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.120510 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" path="/var/lib/kubelet/pods/242a6b0e-c77d-488e-a119-4e22b46baf3f/volumes" Jul 14 14:38:11.121045 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.121021 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" path="/var/lib/kubelet/pods/f9bf4597-894e-41c9-baad-16d24c7308ef/volumes" Jul 14 14:38:11.121240 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.121217 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:38:11.121349 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:11.121254 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:38:11.121523 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.121492 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:38:11.121627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.121539 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" event={"ID":"7e5222cd-dbf1-49ef-9a5d-fd9a347b609e","Type":"ContainerStarted","Data":"6cf3b5a016b59ba77c8771e3fbf5a64ac6a68c154cbbe1c5b1de7faf3f653d49"} Jul 14 14:38:11.121627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.121562 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" event={"ID":"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8","Type":"ContainerStarted","Data":"b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd"} Jul 14 14:38:11.121627 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.121589 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" event={"ID":"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8","Type":"ContainerStarted","Data":"358c8db73af31238263cec549a52db823de9dca1e10b96a1ebc19c963f271109"} Jul 14 14:38:11.121788 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.121627 2550 scope.go:117] "RemoveContainer" containerID="7769518e8da3b0c9fd07f6679c8be57e56aea841c96493d2de513c2215da0a33" Jul 14 14:38:11.123086 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.122881 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" event={"ID":"681b4a73-84d5-4c51-9c70-7608697f0ce4","Type":"ContainerStarted","Data":"042f2ab946e7ac5a4a35e5d313de253d6f76f086a587a2e8466d1e166ef51e9b"} Jul 14 14:38:11.123086 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.122923 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" event={"ID":"681b4a73-84d5-4c51-9c70-7608697f0ce4","Type":"ContainerStarted","Data":"409d57bfb253b6116b9087ef296f022e9c06950cc5bba22f22e975c28d30f883"} Jul 14 14:38:11.123086 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.123015 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" Jul 14 14:38:11.132382 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.132350 2550 scope.go:117] "RemoveContainer" containerID="6df3f2d60bff56b032031a5ad6d416f5fec3e12ed22bed60e447fb444851d00f" Jul 14 14:38:11.139997 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.139935 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rvh" podStartSLOduration=3.045361387 podStartE2EDuration="27.13992063s" totalImagesPullingTime="24.094559243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:11.137715073 +0000 UTC m=+480.572699699" watchObservedRunningTime="2026-07-14 14:38:11.13992063 +0000 UTC m=+480.574905298" Jul 14 14:38:11.140662 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:11.140641 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:38:11.164954 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.164906 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" podStartSLOduration=16.164889381 podStartE2EDuration="16.164889381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:11.162061487 +0000 UTC m=+480.597046111" watchObservedRunningTime="2026-07-14 14:38:11.164889381 +0000 UTC m=+480.599874006" Jul 14 14:38:11.179703 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:11.179654 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" podStartSLOduration=16.179640257 podStartE2EDuration="16.179640257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:11.177695533 +0000 UTC m=+480.612680157" watchObservedRunningTime="2026-07-14 14:38:11.179640257 +0000 UTC m=+480.614624953" Jul 14 14:38:11.179878 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:11.179829 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6cpw\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="f9bf4597-894e-41c9-baad-16d24c7308ef" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6cpw" Jul 14 14:38:11.182044 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:11.182022 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q479t\" is forbidden: User \"system:node:ip-10-0-141-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-194.ec2.internal' and this object" podUID="242a6b0e-c77d-488e-a119-4e22b46baf3f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q479t" Jul 14 14:38:12.913293 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:12.913254 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv"] Jul 14 14:38:14.137013 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:14.136953 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" podUID="d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8" containerName="manager" containerID="cri-o://b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd" gracePeriod=10 Jul 14 14:38:14.386294 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:14.386265 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:38:14.530968 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:14.530936 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jppnp\" (UniqueName: \"kubernetes.io/projected/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-kube-api-access-jppnp\") pod \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\" (UID: \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\") " Jul 14 14:38:14.531154 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:14.531004 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-extensions-socket-volume\") pod \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\" (UID: \"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8\") " Jul 14 14:38:14.531380 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:14.531354 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8" (UID: "d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:38:14.533036 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:14.533007 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-kube-api-access-jppnp" (OuterVolumeSpecName: "kube-api-access-jppnp") pod "d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8" (UID: "d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8"). InnerVolumeSpecName "kube-api-access-jppnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:38:14.631957 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:14.631918 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jppnp\" (UniqueName: \"kubernetes.io/projected/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-kube-api-access-jppnp\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:14.631957 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:14.631951 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8-extensions-socket-volume\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:15.142577 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.142542 2550 generic.go:356] "Generic (PLEG): container finished" podID="d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8" containerID="b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd" exitCode=0 Jul 14 14:38:15.143008 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.142585 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" Jul 14 14:38:15.143008 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.142623 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" event={"ID":"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8","Type":"ContainerDied","Data":"b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd"} Jul 14 14:38:15.143008 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.142676 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv" event={"ID":"d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8","Type":"ContainerDied","Data":"358c8db73af31238263cec549a52db823de9dca1e10b96a1ebc19c963f271109"} Jul 14 14:38:15.143008 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.142698 2550 scope.go:117] "RemoveContainer" containerID="b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd" Jul 14 14:38:15.151224 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.151202 2550 scope.go:117] "RemoveContainer" containerID="b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd" Jul 14 14:38:15.151514 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:15.151497 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd\": container with ID starting with b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd not found: ID does not exist" containerID="b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd" Jul 14 14:38:15.151571 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.151521 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd"} err="failed to get container status \"b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd\": rpc error: code = NotFound desc = could not find container \"b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd\": container with ID starting with b3f5b9abc6e61d808606efc4a0ba902a55beaa96a71851e035515be3abfff3fd not found: ID does not exist" Jul 14 14:38:15.166193 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.166164 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv"] Jul 14 14:38:15.170989 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:15.170947 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xmvxv"] Jul 14 14:38:17.120512 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:17.120483 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8" path="/var/lib/kubelet/pods/d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8/volumes" Jul 14 14:38:22.130789 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:22.130752 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq5l9" Jul 14 14:38:27.155457 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.155397 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fdb746689-l2m7b" podUID="27419db7-a41c-48ce-83de-8dd3b60f9f1c" containerName="console" containerID="cri-o://878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95" gracePeriod=15 Jul 14 14:38:27.403611 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.403582 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:38:27.425088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425006 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-oauth-serving-cert\") pod \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " Jul 14 14:38:27.425088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425055 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-service-ca\") pod \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " Jul 14 14:38:27.425088 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425083 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtmsg\" (UniqueName: \"kubernetes.io/projected/27419db7-a41c-48ce-83de-8dd3b60f9f1c-kube-api-access-gtmsg\") pod \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " Jul 14 14:38:27.425366 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425116 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-oauth-config\") pod \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " Jul 14 14:38:27.425570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425464 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "27419db7-a41c-48ce-83de-8dd3b60f9f1c" (UID: "27419db7-a41c-48ce-83de-8dd3b60f9f1c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:38:27.425570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425473 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-trusted-ca-bundle\") pod \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " Jul 14 14:38:27.425570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425504 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-service-ca" (OuterVolumeSpecName: "service-ca") pod "27419db7-a41c-48ce-83de-8dd3b60f9f1c" (UID: "27419db7-a41c-48ce-83de-8dd3b60f9f1c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:38:27.426070 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425592 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-config\") pod \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " Jul 14 14:38:27.426070 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425724 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-serving-cert\") pod \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\" (UID: \"27419db7-a41c-48ce-83de-8dd3b60f9f1c\") " Jul 14 14:38:27.426070 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.425786 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "27419db7-a41c-48ce-83de-8dd3b60f9f1c" (UID: "27419db7-a41c-48ce-83de-8dd3b60f9f1c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:38:27.426070 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.426008 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-oauth-serving-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:27.426070 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.426025 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-service-ca\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:27.426070 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.426038 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-trusted-ca-bundle\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:27.426469 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.426417 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-config" (OuterVolumeSpecName: "console-config") pod "27419db7-a41c-48ce-83de-8dd3b60f9f1c" (UID: "27419db7-a41c-48ce-83de-8dd3b60f9f1c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:38:27.427821 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.427791 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27419db7-a41c-48ce-83de-8dd3b60f9f1c-kube-api-access-gtmsg" (OuterVolumeSpecName: "kube-api-access-gtmsg") pod "27419db7-a41c-48ce-83de-8dd3b60f9f1c" (UID: "27419db7-a41c-48ce-83de-8dd3b60f9f1c"). InnerVolumeSpecName "kube-api-access-gtmsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:38:27.427931 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.427895 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "27419db7-a41c-48ce-83de-8dd3b60f9f1c" (UID: "27419db7-a41c-48ce-83de-8dd3b60f9f1c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:38:27.428403 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.428384 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "27419db7-a41c-48ce-83de-8dd3b60f9f1c" (UID: "27419db7-a41c-48ce-83de-8dd3b60f9f1c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:38:27.527241 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.527198 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-config\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:27.527241 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.527233 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-serving-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:27.527241 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.527244 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmsg\" (UniqueName: \"kubernetes.io/projected/27419db7-a41c-48ce-83de-8dd3b60f9f1c-kube-api-access-gtmsg\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:27.527241 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:27.527253 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27419db7-a41c-48ce-83de-8dd3b60f9f1c-console-oauth-config\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:28.206503 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.206467 2550 generic.go:356] "Generic (PLEG): container finished" podID="27419db7-a41c-48ce-83de-8dd3b60f9f1c" containerID="878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95" exitCode=0 Jul 14 14:38:28.206913 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.206530 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdb746689-l2m7b" Jul 14 14:38:28.206913 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.206553 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdb746689-l2m7b" event={"ID":"27419db7-a41c-48ce-83de-8dd3b60f9f1c","Type":"ContainerDied","Data":"878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95"} Jul 14 14:38:28.206913 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.206637 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdb746689-l2m7b" event={"ID":"27419db7-a41c-48ce-83de-8dd3b60f9f1c","Type":"ContainerDied","Data":"990629dcb623802329314d1735cb5bd9427e89167c4963037223d96efd550d68"} Jul 14 14:38:28.206913 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.206664 2550 scope.go:117] "RemoveContainer" containerID="878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95" Jul 14 14:38:28.215633 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.215607 2550 scope.go:117] "RemoveContainer" containerID="878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95" Jul 14 14:38:28.215924 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:28.215904 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95\": container with ID starting with 878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95 not found: ID does not exist" containerID="878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95" Jul 14 14:38:28.216016 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.215932 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95"} err="failed to get container status \"878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95\": rpc error: code = NotFound desc = could not find container \"878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95\": container with ID starting with 878f757c02f1c8ade767721815cba9d72ee2b94eacc38c76723b5c2dc9003b95 not found: ID does not exist" Jul 14 14:38:28.231146 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.231116 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdb746689-l2m7b"] Jul 14 14:38:28.235495 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:28.235465 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fdb746689-l2m7b"] Jul 14 14:38:29.121073 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.121043 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27419db7-a41c-48ce-83de-8dd3b60f9f1c" path="/var/lib/kubelet/pods/27419db7-a41c-48ce-83de-8dd3b60f9f1c/volumes" Jul 14 14:38:29.229865 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.229831 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2"] Jul 14 14:38:29.230331 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.230213 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8" containerName="manager" Jul 14 14:38:29.230331 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.230225 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8" containerName="manager" Jul 14 14:38:29.230331 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.230234 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27419db7-a41c-48ce-83de-8dd3b60f9f1c" containerName="console" Jul 14 14:38:29.230331 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.230240 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="27419db7-a41c-48ce-83de-8dd3b60f9f1c" containerName="console" Jul 14 14:38:29.230331 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.230300 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="27419db7-a41c-48ce-83de-8dd3b60f9f1c" containerName="console" Jul 14 14:38:29.230331 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.230310 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b1e4b4-ad38-418c-96f5-a9ad6a68a6e8" containerName="manager" Jul 14 14:38:29.234838 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.234813 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.238204 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.238185 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pvpvp\"" Jul 14 14:38:29.239026 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239001 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.239113 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239055 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.239113 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239087 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.239204 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239112 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.239204 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239142 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.239204 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239178 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252f0ccf-34b0-4ab7-9641-c1b973022da1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.239308 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239238 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.239308 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239276 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.239308 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.239302 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2md\" (UniqueName: \"kubernetes.io/projected/252f0ccf-34b0-4ab7-9641-c1b973022da1-kube-api-access-6h2md\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.244994 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.244953 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2"] Jul 14 14:38:29.339818 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.339783 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340038 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.339831 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340038 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.339857 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340038 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.339879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340038 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.339913 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340038 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.339949 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252f0ccf-34b0-4ab7-9641-c1b973022da1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340038 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.339998 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340674 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.340045 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340674 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.340072 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2md\" (UniqueName: \"kubernetes.io/projected/252f0ccf-34b0-4ab7-9641-c1b973022da1-kube-api-access-6h2md\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340674 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.340270 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340674 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.340279 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340674 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.340442 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340674 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.340568 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.340935 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.340706 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252f0ccf-34b0-4ab7-9641-c1b973022da1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.342552 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.342529 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.342813 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.342792 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.347893 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.347871 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252f0ccf-34b0-4ab7-9641-c1b973022da1-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.348020 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.347875 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2md\" (UniqueName: \"kubernetes.io/projected/252f0ccf-34b0-4ab7-9641-c1b973022da1-kube-api-access-6h2md\") pod \"maas-default-gateway-openshift-default-5df9b4f89-wvnd2\" (UID: \"252f0ccf-34b0-4ab7-9641-c1b973022da1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.545482 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.545436 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:29.715290 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.715246 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2"] Jul 14 14:38:29.718815 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:38:29.718781 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252f0ccf_34b0_4ab7_9641_c1b973022da1.slice/crio-5c6a1935fb4028b71e76ba0488cb9e5bff91c129d7a46df488d4fae4a8a593c9 WatchSource:0}: Error finding container 5c6a1935fb4028b71e76ba0488cb9e5bff91c129d7a46df488d4fae4a8a593c9: Status 404 returned error can't find the container with id 5c6a1935fb4028b71e76ba0488cb9e5bff91c129d7a46df488d4fae4a8a593c9 Jul 14 14:38:29.721420 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.721385 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:38:29.721501 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.721453 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:38:29.721501 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:29.721483 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:38:30.220788 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:30.220696 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" event={"ID":"252f0ccf-34b0-4ab7-9641-c1b973022da1","Type":"ContainerStarted","Data":"4c23011a5fa332eccdd01963ce500ab7a80355a692b24e6dca36d98b5b21351d"} Jul 14 14:38:30.220788 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:30.220786 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" event={"ID":"252f0ccf-34b0-4ab7-9641-c1b973022da1","Type":"ContainerStarted","Data":"5c6a1935fb4028b71e76ba0488cb9e5bff91c129d7a46df488d4fae4a8a593c9"} Jul 14 14:38:30.243611 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:30.243554 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" podStartSLOduration=1.243538437 podStartE2EDuration="1.243538437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:30.241143975 +0000 UTC m=+499.676128593" watchObservedRunningTime="2026-07-14 14:38:30.243538437 +0000 UTC m=+499.678523139" Jul 14 14:38:30.546342 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:30.546305 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:30.551042 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:30.551013 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:31.225459 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.225424 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:31.226803 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.226779 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:38:31.242245 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.242212 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8rz68"] Jul 14 14:38:31.280373 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.280340 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8rz68"] Jul 14 14:38:31.280829 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.280470 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:31.283291 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.283269 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 14:38:31.337504 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.337473 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8rz68"] Jul 14 14:38:31.361100 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.361069 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1216f6a-2e6a-465d-972d-9fd802912025-config-file\") pod \"limitador-limitador-64c94fd64f-8rz68\" (UID: \"b1216f6a-2e6a-465d-972d-9fd802912025\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:31.361282 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.361151 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncksh\" (UniqueName: \"kubernetes.io/projected/b1216f6a-2e6a-465d-972d-9fd802912025-kube-api-access-ncksh\") pod \"limitador-limitador-64c94fd64f-8rz68\" (UID: \"b1216f6a-2e6a-465d-972d-9fd802912025\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:31.461906 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.461868 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1216f6a-2e6a-465d-972d-9fd802912025-config-file\") pod \"limitador-limitador-64c94fd64f-8rz68\" (UID: \"b1216f6a-2e6a-465d-972d-9fd802912025\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:31.462087 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.461982 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncksh\" (UniqueName: \"kubernetes.io/projected/b1216f6a-2e6a-465d-972d-9fd802912025-kube-api-access-ncksh\") pod \"limitador-limitador-64c94fd64f-8rz68\" (UID: \"b1216f6a-2e6a-465d-972d-9fd802912025\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:31.462537 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.462518 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1216f6a-2e6a-465d-972d-9fd802912025-config-file\") pod \"limitador-limitador-64c94fd64f-8rz68\" (UID: \"b1216f6a-2e6a-465d-972d-9fd802912025\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:31.472289 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.472255 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncksh\" (UniqueName: \"kubernetes.io/projected/b1216f6a-2e6a-465d-972d-9fd802912025-kube-api-access-ncksh\") pod \"limitador-limitador-64c94fd64f-8rz68\" (UID: \"b1216f6a-2e6a-465d-972d-9fd802912025\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:31.592438 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.592401 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:31.829385 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:31.829360 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8rz68"] Jul 14 14:38:31.831738 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:38:31.831707 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1216f6a_2e6a_465d_972d_9fd802912025.slice/crio-9f14a813bb6f3fc92ba5dd81f08a96686e8f44563acc97a6a53091890386b223 WatchSource:0}: Error finding container 9f14a813bb6f3fc92ba5dd81f08a96686e8f44563acc97a6a53091890386b223: Status 404 returned error can't find the container with id 9f14a813bb6f3fc92ba5dd81f08a96686e8f44563acc97a6a53091890386b223 Jul 14 14:38:32.123167 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.123134 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-jbqzr"] Jul 14 14:38:32.162451 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.162412 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jbqzr"] Jul 14 14:38:32.162648 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.162517 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jbqzr" Jul 14 14:38:32.165295 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.165270 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9gjbq\"" Jul 14 14:38:32.231653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.231619 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" event={"ID":"b1216f6a-2e6a-465d-972d-9fd802912025","Type":"ContainerStarted","Data":"9f14a813bb6f3fc92ba5dd81f08a96686e8f44563acc97a6a53091890386b223"} Jul 14 14:38:32.268466 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.268430 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fch\" (UniqueName: \"kubernetes.io/projected/ac8820c0-9639-4a8c-9f9e-d61aa362256b-kube-api-access-j7fch\") pod \"authorino-58cd955675-jbqzr\" (UID: \"ac8820c0-9639-4a8c-9f9e-d61aa362256b\") " pod="kuadrant-system/authorino-58cd955675-jbqzr" Jul 14 14:38:32.369275 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.369227 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fch\" (UniqueName: \"kubernetes.io/projected/ac8820c0-9639-4a8c-9f9e-d61aa362256b-kube-api-access-j7fch\") pod \"authorino-58cd955675-jbqzr\" (UID: \"ac8820c0-9639-4a8c-9f9e-d61aa362256b\") " pod="kuadrant-system/authorino-58cd955675-jbqzr" Jul 14 14:38:32.378955 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.378889 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fch\" (UniqueName: \"kubernetes.io/projected/ac8820c0-9639-4a8c-9f9e-d61aa362256b-kube-api-access-j7fch\") pod \"authorino-58cd955675-jbqzr\" (UID: \"ac8820c0-9639-4a8c-9f9e-d61aa362256b\") " pod="kuadrant-system/authorino-58cd955675-jbqzr" Jul 14 14:38:32.472629 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.472583 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jbqzr" Jul 14 14:38:32.686033 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:32.685619 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jbqzr"] Jul 14 14:38:32.688577 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:38:32.688550 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac8820c0_9639_4a8c_9f9e_d61aa362256b.slice/crio-8abb99225180bdea28d27252557aad19fa59b8e28692c913ad8be21831a3a3f6 WatchSource:0}: Error finding container 8abb99225180bdea28d27252557aad19fa59b8e28692c913ad8be21831a3a3f6: Status 404 returned error can't find the container with id 8abb99225180bdea28d27252557aad19fa59b8e28692c913ad8be21831a3a3f6 Jul 14 14:38:33.239809 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:33.239765 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jbqzr" event={"ID":"ac8820c0-9639-4a8c-9f9e-d61aa362256b","Type":"ContainerStarted","Data":"8abb99225180bdea28d27252557aad19fa59b8e28692c913ad8be21831a3a3f6"} Jul 14 14:38:37.073748 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:37.073700 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jbqzr"] Jul 14 14:38:39.272355 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.272319 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" event={"ID":"b1216f6a-2e6a-465d-972d-9fd802912025","Type":"ContainerStarted","Data":"ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8"} Jul 14 14:38:39.272837 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.272413 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:39.273993 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.273952 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jbqzr" event={"ID":"ac8820c0-9639-4a8c-9f9e-d61aa362256b","Type":"ContainerStarted","Data":"b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957"} Jul 14 14:38:39.274108 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.273956 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-jbqzr" podUID="ac8820c0-9639-4a8c-9f9e-d61aa362256b" containerName="authorino" containerID="cri-o://b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957" gracePeriod=30 Jul 14 14:38:39.290093 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.290037 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" podStartSLOduration=1.846716554 podStartE2EDuration="8.290014246s" totalImagesPullingTime="6.443297692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:39.289605056 +0000 UTC m=+508.724589680" watchObservedRunningTime="2026-07-14 14:38:39.290014246 +0000 UTC m=+508.724998871" Jul 14 14:38:39.304092 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.304048 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-jbqzr" podStartSLOduration=1.703038142 podStartE2EDuration="7.304030116s" totalImagesPullingTime="5.600991974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:39.303894119 +0000 UTC m=+508.738878744" watchObservedRunningTime="2026-07-14 14:38:39.304030116 +0000 UTC m=+508.739014739" Jul 14 14:38:39.584303 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.584272 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jbqzr" Jul 14 14:38:39.640696 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.640660 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7fch\" (UniqueName: \"kubernetes.io/projected/ac8820c0-9639-4a8c-9f9e-d61aa362256b-kube-api-access-j7fch\") pod \"ac8820c0-9639-4a8c-9f9e-d61aa362256b\" (UID: \"ac8820c0-9639-4a8c-9f9e-d61aa362256b\") " Jul 14 14:38:39.642733 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.642705 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8820c0-9639-4a8c-9f9e-d61aa362256b-kube-api-access-j7fch" (OuterVolumeSpecName: "kube-api-access-j7fch") pod "ac8820c0-9639-4a8c-9f9e-d61aa362256b" (UID: "ac8820c0-9639-4a8c-9f9e-d61aa362256b"). InnerVolumeSpecName "kube-api-access-j7fch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:38:39.741845 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:39.741814 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fch\" (UniqueName: \"kubernetes.io/projected/ac8820c0-9639-4a8c-9f9e-d61aa362256b-kube-api-access-j7fch\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:40.280186 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.280147 2550 generic.go:356] "Generic (PLEG): container finished" podID="ac8820c0-9639-4a8c-9f9e-d61aa362256b" containerID="b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957" exitCode=0 Jul 14 14:38:40.280634 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.280276 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jbqzr" Jul 14 14:38:40.280634 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.280293 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jbqzr" event={"ID":"ac8820c0-9639-4a8c-9f9e-d61aa362256b","Type":"ContainerDied","Data":"b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957"} Jul 14 14:38:40.280634 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.280347 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jbqzr" event={"ID":"ac8820c0-9639-4a8c-9f9e-d61aa362256b","Type":"ContainerDied","Data":"8abb99225180bdea28d27252557aad19fa59b8e28692c913ad8be21831a3a3f6"} Jul 14 14:38:40.280634 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.280371 2550 scope.go:117] "RemoveContainer" containerID="b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957" Jul 14 14:38:40.290141 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.290120 2550 scope.go:117] "RemoveContainer" containerID="b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957" Jul 14 14:38:40.290412 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:40.290395 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957\": container with ID starting with b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957 not found: ID does not exist" containerID="b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957" Jul 14 14:38:40.290480 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.290422 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957"} err="failed to get container status \"b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957\": rpc error: code = NotFound desc = could not find container \"b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957\": container with ID starting with b151afae09373ed416c66659e81de7afae225bdfd2bfbf256c13332363f9d957 not found: ID does not exist" Jul 14 14:38:40.305273 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.305245 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jbqzr"] Jul 14 14:38:40.310528 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:40.310502 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-jbqzr"] Jul 14 14:38:41.120536 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:41.120500 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac8820c0-9639-4a8c-9f9e-d61aa362256b" path="/var/lib/kubelet/pods/ac8820c0-9639-4a8c-9f9e-d61aa362256b/volumes" Jul 14 14:38:46.149821 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.149783 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8rz68"] Jul 14 14:38:46.150259 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.150010 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" podUID="b1216f6a-2e6a-465d-972d-9fd802912025" containerName="limitador" containerID="cri-o://ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8" gracePeriod=30 Jul 14 14:38:46.152070 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.152041 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:46.701219 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.701194 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:46.802691 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.802651 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1216f6a-2e6a-465d-972d-9fd802912025-config-file\") pod \"b1216f6a-2e6a-465d-972d-9fd802912025\" (UID: \"b1216f6a-2e6a-465d-972d-9fd802912025\") " Jul 14 14:38:46.802900 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.802879 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncksh\" (UniqueName: \"kubernetes.io/projected/b1216f6a-2e6a-465d-972d-9fd802912025-kube-api-access-ncksh\") pod \"b1216f6a-2e6a-465d-972d-9fd802912025\" (UID: \"b1216f6a-2e6a-465d-972d-9fd802912025\") " Jul 14 14:38:46.803061 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.803036 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1216f6a-2e6a-465d-972d-9fd802912025-config-file" (OuterVolumeSpecName: "config-file") pod "b1216f6a-2e6a-465d-972d-9fd802912025" (UID: "b1216f6a-2e6a-465d-972d-9fd802912025"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:38:46.803201 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.803186 2550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1216f6a-2e6a-465d-972d-9fd802912025-config-file\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:46.805286 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.805257 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1216f6a-2e6a-465d-972d-9fd802912025-kube-api-access-ncksh" (OuterVolumeSpecName: "kube-api-access-ncksh") pod "b1216f6a-2e6a-465d-972d-9fd802912025" (UID: "b1216f6a-2e6a-465d-972d-9fd802912025"). InnerVolumeSpecName "kube-api-access-ncksh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:38:46.903596 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:46.903553 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncksh\" (UniqueName: \"kubernetes.io/projected/b1216f6a-2e6a-465d-972d-9fd802912025-kube-api-access-ncksh\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:38:47.314950 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.314914 2550 generic.go:356] "Generic (PLEG): container finished" podID="b1216f6a-2e6a-465d-972d-9fd802912025" containerID="ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8" exitCode=0 Jul 14 14:38:47.315373 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.314981 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" Jul 14 14:38:47.315373 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.315006 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" event={"ID":"b1216f6a-2e6a-465d-972d-9fd802912025","Type":"ContainerDied","Data":"ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8"} Jul 14 14:38:47.315373 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.315063 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8rz68" event={"ID":"b1216f6a-2e6a-465d-972d-9fd802912025","Type":"ContainerDied","Data":"9f14a813bb6f3fc92ba5dd81f08a96686e8f44563acc97a6a53091890386b223"} Jul 14 14:38:47.315373 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.315087 2550 scope.go:117] "RemoveContainer" containerID="ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8" Jul 14 14:38:47.323629 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.323611 2550 scope.go:117] "RemoveContainer" containerID="ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8" Jul 14 14:38:47.323868 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:38:47.323854 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8\": container with ID starting with ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8 not found: ID does not exist" containerID="ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8" Jul 14 14:38:47.323921 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.323876 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8"} err="failed to get container status \"ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8\": rpc error: code = NotFound desc = could not find container \"ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8\": container with ID starting with ab752581bf7da341894d857ecf9c2ebfef526ce173dcc3b0d36e9dad939e9bc8 not found: ID does not exist" Jul 14 14:38:47.333250 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.333227 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8rz68"] Jul 14 14:38:47.336647 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.336625 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8rz68"] Jul 14 14:38:47.879563 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.879526 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt"] Jul 14 14:38:47.879873 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.879862 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1216f6a-2e6a-465d-972d-9fd802912025" containerName="limitador" Jul 14 14:38:47.879931 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.879875 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1216f6a-2e6a-465d-972d-9fd802912025" containerName="limitador" Jul 14 14:38:47.879931 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.879889 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8820c0-9639-4a8c-9f9e-d61aa362256b" containerName="authorino" Jul 14 14:38:47.879931 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.879894 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8820c0-9639-4a8c-9f9e-d61aa362256b" containerName="authorino" Jul 14 14:38:47.880051 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.879943 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1216f6a-2e6a-465d-972d-9fd802912025" containerName="limitador" Jul 14 14:38:47.880051 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.879954 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8820c0-9639-4a8c-9f9e-d61aa362256b" containerName="authorino" Jul 14 14:38:47.884224 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.884202 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:47.887152 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.887128 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 14 14:38:47.887279 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.887129 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 14:38:47.887279 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.887159 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fs9xx\"" Jul 14 14:38:47.888291 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.888276 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 14 14:38:47.891819 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:47.891799 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt"] Jul 14 14:38:48.012512 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:48.012474 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/060c90ba-b8e2-4746-a0f4-bf814683fbdf-data\") pod \"postgres-69fb4594cf-bdwlt\" (UID: \"060c90ba-b8e2-4746-a0f4-bf814683fbdf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:48.012695 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:48.012529 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsk9f\" (UniqueName: \"kubernetes.io/projected/060c90ba-b8e2-4746-a0f4-bf814683fbdf-kube-api-access-vsk9f\") pod \"postgres-69fb4594cf-bdwlt\" (UID: \"060c90ba-b8e2-4746-a0f4-bf814683fbdf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:48.113107 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:48.113064 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsk9f\" (UniqueName: \"kubernetes.io/projected/060c90ba-b8e2-4746-a0f4-bf814683fbdf-kube-api-access-vsk9f\") pod \"postgres-69fb4594cf-bdwlt\" (UID: \"060c90ba-b8e2-4746-a0f4-bf814683fbdf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:48.113331 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:48.113216 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/060c90ba-b8e2-4746-a0f4-bf814683fbdf-data\") pod \"postgres-69fb4594cf-bdwlt\" (UID: \"060c90ba-b8e2-4746-a0f4-bf814683fbdf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:48.113595 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:48.113575 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/060c90ba-b8e2-4746-a0f4-bf814683fbdf-data\") pod \"postgres-69fb4594cf-bdwlt\" (UID: \"060c90ba-b8e2-4746-a0f4-bf814683fbdf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:48.123414 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:48.123384 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsk9f\" (UniqueName: \"kubernetes.io/projected/060c90ba-b8e2-4746-a0f4-bf814683fbdf-kube-api-access-vsk9f\") pod \"postgres-69fb4594cf-bdwlt\" (UID: \"060c90ba-b8e2-4746-a0f4-bf814683fbdf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:48.194580 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:48.194484 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:48.330711 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:48.330686 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt"] Jul 14 14:38:48.332792 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:38:48.332760 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060c90ba_b8e2_4746_a0f4_bf814683fbdf.slice/crio-6bf98dbc4b5b1153f857be6bcb1350bdc537d32db03857083c484f26f661825b WatchSource:0}: Error finding container 6bf98dbc4b5b1153f857be6bcb1350bdc537d32db03857083c484f26f661825b: Status 404 returned error can't find the container with id 6bf98dbc4b5b1153f857be6bcb1350bdc537d32db03857083c484f26f661825b Jul 14 14:38:49.127695 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:49.127652 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1216f6a-2e6a-465d-972d-9fd802912025" path="/var/lib/kubelet/pods/b1216f6a-2e6a-465d-972d-9fd802912025/volumes" Jul 14 14:38:49.329861 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:49.329819 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" event={"ID":"060c90ba-b8e2-4746-a0f4-bf814683fbdf","Type":"ContainerStarted","Data":"6bf98dbc4b5b1153f857be6bcb1350bdc537d32db03857083c484f26f661825b"} Jul 14 14:38:53.355074 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:53.354989 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" event={"ID":"060c90ba-b8e2-4746-a0f4-bf814683fbdf","Type":"ContainerStarted","Data":"0931680632ba067cd4cf49bf64a1ccfc11a874b25769a175640565a20b827d49"} Jul 14 14:38:53.355437 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:53.355083 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:53.372477 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:53.372429 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" podStartSLOduration=1.693954299 podStartE2EDuration="6.372413292s" totalImagesPullingTime="4.678458993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:53.370645665 +0000 UTC m=+522.805630287" watchObservedRunningTime="2026-07-14 14:38:53.372413292 +0000 UTC m=+522.807397913" Jul 14 14:38:59.389289 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:59.389260 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bdwlt" Jul 14 14:38:59.935614 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:59.935575 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9n6dj"] Jul 14 14:38:59.941768 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:59.941741 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" Jul 14 14:38:59.944597 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:59.944570 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9gjbq\"" Jul 14 14:38:59.947287 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:38:59.947257 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9n6dj"] Jul 14 14:39:00.025056 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.025019 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhrt\" (UniqueName: \"kubernetes.io/projected/38884dc4-a4b2-40ec-ba09-da52ba81c659-kube-api-access-jmhrt\") pod \"authorino-5d4b67c446-9n6dj\" (UID: \"38884dc4-a4b2-40ec-ba09-da52ba81c659\") " pod="kuadrant-system/authorino-5d4b67c446-9n6dj" Jul 14 14:39:00.126510 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.126473 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhrt\" (UniqueName: \"kubernetes.io/projected/38884dc4-a4b2-40ec-ba09-da52ba81c659-kube-api-access-jmhrt\") pod \"authorino-5d4b67c446-9n6dj\" (UID: \"38884dc4-a4b2-40ec-ba09-da52ba81c659\") " pod="kuadrant-system/authorino-5d4b67c446-9n6dj" Jul 14 14:39:00.134469 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.134430 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhrt\" (UniqueName: \"kubernetes.io/projected/38884dc4-a4b2-40ec-ba09-da52ba81c659-kube-api-access-jmhrt\") pod \"authorino-5d4b67c446-9n6dj\" (UID: \"38884dc4-a4b2-40ec-ba09-da52ba81c659\") " pod="kuadrant-system/authorino-5d4b67c446-9n6dj" Jul 14 14:39:00.177473 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.177438 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9n6dj"] Jul 14 14:39:00.177701 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.177687 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" Jul 14 14:39:00.206425 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.206382 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-544f849cd-twvm8"] Jul 14 14:39:00.211667 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.211642 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544f849cd-twvm8" Jul 14 14:39:00.217066 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.217034 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544f849cd-twvm8"] Jul 14 14:39:00.317223 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.317152 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9n6dj"] Jul 14 14:39:00.319796 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:39:00.319773 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38884dc4_a4b2_40ec_ba09_da52ba81c659.slice/crio-cc020ccf13b52538983c4bb3ac07360373b676031546eca99a9949523779ea02 WatchSource:0}: Error finding container cc020ccf13b52538983c4bb3ac07360373b676031546eca99a9949523779ea02: Status 404 returned error can't find the container with id cc020ccf13b52538983c4bb3ac07360373b676031546eca99a9949523779ea02 Jul 14 14:39:00.327913 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.327886 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c455l\" (UniqueName: \"kubernetes.io/projected/70c45567-1594-4c81-8a10-5cd26b43d5f4-kube-api-access-c455l\") pod \"authorino-544f849cd-twvm8\" (UID: \"70c45567-1594-4c81-8a10-5cd26b43d5f4\") " pod="kuadrant-system/authorino-544f849cd-twvm8" Jul 14 14:39:00.388231 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.388199 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" event={"ID":"38884dc4-a4b2-40ec-ba09-da52ba81c659","Type":"ContainerStarted","Data":"cc020ccf13b52538983c4bb3ac07360373b676031546eca99a9949523779ea02"} Jul 14 14:39:00.428665 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.428623 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c455l\" (UniqueName: \"kubernetes.io/projected/70c45567-1594-4c81-8a10-5cd26b43d5f4-kube-api-access-c455l\") pod \"authorino-544f849cd-twvm8\" (UID: \"70c45567-1594-4c81-8a10-5cd26b43d5f4\") " pod="kuadrant-system/authorino-544f849cd-twvm8" Jul 14 14:39:00.437402 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.437371 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c455l\" (UniqueName: \"kubernetes.io/projected/70c45567-1594-4c81-8a10-5cd26b43d5f4-kube-api-access-c455l\") pod \"authorino-544f849cd-twvm8\" (UID: \"70c45567-1594-4c81-8a10-5cd26b43d5f4\") " pod="kuadrant-system/authorino-544f849cd-twvm8" Jul 14 14:39:00.477303 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.477222 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-544f849cd-twvm8"] Jul 14 14:39:00.477487 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.477475 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544f849cd-twvm8" Jul 14 14:39:00.505223 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.505165 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f647cb5d6-b2fr2"] Jul 14 14:39:00.509993 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.509958 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:39:00.512750 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.512725 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 14:39:00.518657 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.518633 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f647cb5d6-b2fr2"] Jul 14 14:39:00.613858 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.613829 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-544f849cd-twvm8"] Jul 14 14:39:00.617529 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:39:00.617500 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70c45567_1594_4c81_8a10_5cd26b43d5f4.slice/crio-c436baaf3c8cab52eb21ca039d0f206e5812a025b3a7fbe30d0899e19121d8db WatchSource:0}: Error finding container c436baaf3c8cab52eb21ca039d0f206e5812a025b3a7fbe30d0899e19121d8db: Status 404 returned error can't find the container with id c436baaf3c8cab52eb21ca039d0f206e5812a025b3a7fbe30d0899e19121d8db Jul 14 14:39:00.630385 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.630350 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvfr\" (UniqueName: \"kubernetes.io/projected/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-kube-api-access-lsvfr\") pod \"authorino-6f647cb5d6-b2fr2\" (UID: \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\") " pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:39:00.630494 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.630413 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-tls-cert\") pod \"authorino-6f647cb5d6-b2fr2\" (UID: \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\") " pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:39:00.737316 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.732103 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvfr\" (UniqueName: \"kubernetes.io/projected/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-kube-api-access-lsvfr\") pod \"authorino-6f647cb5d6-b2fr2\" (UID: \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\") " pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:39:00.737316 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.732196 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-tls-cert\") pod \"authorino-6f647cb5d6-b2fr2\" (UID: \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\") " pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:39:00.737316 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.735402 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-tls-cert\") pod \"authorino-6f647cb5d6-b2fr2\" (UID: \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\") " pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:39:00.740708 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.740680 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvfr\" (UniqueName: \"kubernetes.io/projected/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-kube-api-access-lsvfr\") pod \"authorino-6f647cb5d6-b2fr2\" (UID: \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\") " pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:39:00.820667 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.820623 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:39:00.960646 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:00.960621 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f647cb5d6-b2fr2"] Jul 14 14:39:00.963041 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:39:00.963008 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19b4ee44_4c9c_48f6_a24f_5ebee5d8f865.slice/crio-9a4ae9028c8c3054b9e7bdab1d80189a47413b6dde957cd895cf9569bcc20ba5 WatchSource:0}: Error finding container 9a4ae9028c8c3054b9e7bdab1d80189a47413b6dde957cd895cf9569bcc20ba5: Status 404 returned error can't find the container with id 9a4ae9028c8c3054b9e7bdab1d80189a47413b6dde957cd895cf9569bcc20ba5 Jul 14 14:39:01.393851 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.393818 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" event={"ID":"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865","Type":"ContainerStarted","Data":"9a4ae9028c8c3054b9e7bdab1d80189a47413b6dde957cd895cf9569bcc20ba5"} Jul 14 14:39:01.395393 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.395225 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" event={"ID":"38884dc4-a4b2-40ec-ba09-da52ba81c659","Type":"ContainerStarted","Data":"279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655"} Jul 14 14:39:01.395393 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.395279 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" podUID="38884dc4-a4b2-40ec-ba09-da52ba81c659" containerName="authorino" containerID="cri-o://279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655" gracePeriod=30 Jul 14 14:39:01.397152 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.397111 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544f849cd-twvm8" event={"ID":"70c45567-1594-4c81-8a10-5cd26b43d5f4","Type":"ContainerStarted","Data":"6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8"} Jul 14 14:39:01.397263 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.397157 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544f849cd-twvm8" event={"ID":"70c45567-1594-4c81-8a10-5cd26b43d5f4","Type":"ContainerStarted","Data":"c436baaf3c8cab52eb21ca039d0f206e5812a025b3a7fbe30d0899e19121d8db"} Jul 14 14:39:01.397263 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.397216 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-544f849cd-twvm8" podUID="70c45567-1594-4c81-8a10-5cd26b43d5f4" containerName="authorino" containerID="cri-o://6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8" gracePeriod=30 Jul 14 14:39:01.411668 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.411616 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" podStartSLOduration=1.884538396 podStartE2EDuration="2.411600524s" totalImagesPullingTime="527.062128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:01.410074863 +0000 UTC m=+530.845059488" watchObservedRunningTime="2026-07-14 14:39:01.411600524 +0000 UTC m=+530.846585146" Jul 14 14:39:01.427056 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.426943 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-544f849cd-twvm8" podStartSLOduration=0.943677354 podStartE2EDuration="1.426927501s" totalImagesPullingTime="483.250147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:01.426108294 +0000 UTC m=+530.861092921" watchObservedRunningTime="2026-07-14 14:39:01.426927501 +0000 UTC m=+530.861912125" Jul 14 14:39:01.690687 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.690652 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" Jul 14 14:39:01.706267 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.706234 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544f849cd-twvm8" Jul 14 14:39:01.841795 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.841757 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmhrt\" (UniqueName: \"kubernetes.io/projected/38884dc4-a4b2-40ec-ba09-da52ba81c659-kube-api-access-jmhrt\") pod \"38884dc4-a4b2-40ec-ba09-da52ba81c659\" (UID: \"38884dc4-a4b2-40ec-ba09-da52ba81c659\") " Jul 14 14:39:01.842000 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.841850 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c455l\" (UniqueName: \"kubernetes.io/projected/70c45567-1594-4c81-8a10-5cd26b43d5f4-kube-api-access-c455l\") pod \"70c45567-1594-4c81-8a10-5cd26b43d5f4\" (UID: \"70c45567-1594-4c81-8a10-5cd26b43d5f4\") " Jul 14 14:39:01.843840 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.843806 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38884dc4-a4b2-40ec-ba09-da52ba81c659-kube-api-access-jmhrt" (OuterVolumeSpecName: "kube-api-access-jmhrt") pod "38884dc4-a4b2-40ec-ba09-da52ba81c659" (UID: "38884dc4-a4b2-40ec-ba09-da52ba81c659"). InnerVolumeSpecName "kube-api-access-jmhrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:39:01.843984 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.843864 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c45567-1594-4c81-8a10-5cd26b43d5f4-kube-api-access-c455l" (OuterVolumeSpecName: "kube-api-access-c455l") pod "70c45567-1594-4c81-8a10-5cd26b43d5f4" (UID: "70c45567-1594-4c81-8a10-5cd26b43d5f4"). InnerVolumeSpecName "kube-api-access-c455l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:39:01.943161 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.943070 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c455l\" (UniqueName: \"kubernetes.io/projected/70c45567-1594-4c81-8a10-5cd26b43d5f4-kube-api-access-c455l\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:39:01.943161 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:01.943101 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmhrt\" (UniqueName: \"kubernetes.io/projected/38884dc4-a4b2-40ec-ba09-da52ba81c659-kube-api-access-jmhrt\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:39:02.402977 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.402919 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" event={"ID":"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865","Type":"ContainerStarted","Data":"0ca6368ca64975067d6dff714d32f4c485bfcf57277aa6723db83a355233bbcb"} Jul 14 14:39:02.404253 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.404226 2550 generic.go:356] "Generic (PLEG): container finished" podID="38884dc4-a4b2-40ec-ba09-da52ba81c659" containerID="279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655" exitCode=0 Jul 14 14:39:02.404392 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.404258 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" Jul 14 14:39:02.404392 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.404307 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" event={"ID":"38884dc4-a4b2-40ec-ba09-da52ba81c659","Type":"ContainerDied","Data":"279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655"} Jul 14 14:39:02.404392 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.404336 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9n6dj" event={"ID":"38884dc4-a4b2-40ec-ba09-da52ba81c659","Type":"ContainerDied","Data":"cc020ccf13b52538983c4bb3ac07360373b676031546eca99a9949523779ea02"} Jul 14 14:39:02.404392 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.404351 2550 scope.go:117] "RemoveContainer" containerID="279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655" Jul 14 14:39:02.405657 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.405637 2550 generic.go:356] "Generic (PLEG): container finished" podID="70c45567-1594-4c81-8a10-5cd26b43d5f4" containerID="6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8" exitCode=0 Jul 14 14:39:02.405762 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.405678 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544f849cd-twvm8" Jul 14 14:39:02.405762 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.405715 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544f849cd-twvm8" event={"ID":"70c45567-1594-4c81-8a10-5cd26b43d5f4","Type":"ContainerDied","Data":"6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8"} Jul 14 14:39:02.405762 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.405755 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544f849cd-twvm8" event={"ID":"70c45567-1594-4c81-8a10-5cd26b43d5f4","Type":"ContainerDied","Data":"c436baaf3c8cab52eb21ca039d0f206e5812a025b3a7fbe30d0899e19121d8db"} Jul 14 14:39:02.414202 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.414181 2550 scope.go:117] "RemoveContainer" containerID="279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655" Jul 14 14:39:02.414488 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:39:02.414469 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655\": container with ID starting with 279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655 not found: ID does not exist" containerID="279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655" Jul 14 14:39:02.414563 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.414500 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655"} err="failed to get container status \"279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655\": rpc error: code = NotFound desc = could not find container \"279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655\": container with ID starting with 279a5c0f631ede1bfefdd38494db774d6dacc90602653f47fe803b8bc3225655 not found: ID does not exist" Jul 14 14:39:02.414563 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.414523 2550 scope.go:117] "RemoveContainer" containerID="6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8" Jul 14 14:39:02.420705 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.420656 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" podStartSLOduration=2.030907947 podStartE2EDuration="2.420639789s" totalImagesPullingTime="389.731842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:02.420583522 +0000 UTC m=+531.855568158" watchObservedRunningTime="2026-07-14 14:39:02.420639789 +0000 UTC m=+531.855624418" Jul 14 14:39:02.422595 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.422579 2550 scope.go:117] "RemoveContainer" containerID="6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8" Jul 14 14:39:02.422847 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:39:02.422827 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8\": container with ID starting with 6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8 not found: ID does not exist" containerID="6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8" Jul 14 14:39:02.422907 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.422858 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8"} err="failed to get container status \"6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8\": rpc error: code = NotFound desc = could not find container \"6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8\": container with ID starting with 6018eea854b7a8caafdf99bf6998f7f9a74792275971ad2d17facae4479c57c8 not found: ID does not exist" Jul 14 14:39:02.437817 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.437791 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-544f849cd-twvm8"] Jul 14 14:39:02.447424 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.447397 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-544f849cd-twvm8"] Jul 14 14:39:02.461865 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.461834 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9n6dj"] Jul 14 14:39:02.465304 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:02.465277 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9n6dj"] Jul 14 14:39:03.120091 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:03.120016 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38884dc4-a4b2-40ec-ba09-da52ba81c659" path="/var/lib/kubelet/pods/38884dc4-a4b2-40ec-ba09-da52ba81c659/volumes" Jul 14 14:39:03.125844 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:03.120714 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c45567-1594-4c81-8a10-5cd26b43d5f4" path="/var/lib/kubelet/pods/70c45567-1594-4c81-8a10-5cd26b43d5f4/volumes" Jul 14 14:39:24.756624 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.756586 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt"] Jul 14 14:39:24.757059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.756926 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38884dc4-a4b2-40ec-ba09-da52ba81c659" containerName="authorino" Jul 14 14:39:24.757059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.756938 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="38884dc4-a4b2-40ec-ba09-da52ba81c659" containerName="authorino" Jul 14 14:39:24.757059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.756952 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c45567-1594-4c81-8a10-5cd26b43d5f4" containerName="authorino" Jul 14 14:39:24.757059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.756958 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c45567-1594-4c81-8a10-5cd26b43d5f4" containerName="authorino" Jul 14 14:39:24.757059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.757057 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="38884dc4-a4b2-40ec-ba09-da52ba81c659" containerName="authorino" Jul 14 14:39:24.757253 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.757067 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c45567-1594-4c81-8a10-5cd26b43d5f4" containerName="authorino" Jul 14 14:39:24.762061 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.762042 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:24.764782 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.764760 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 14:39:24.764902 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.764883 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fj6sv\"" Jul 14 14:39:24.779374 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.779350 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt"] Jul 14 14:39:24.789659 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.789633 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-xbmhh"] Jul 14 14:39:24.791994 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.791954 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:24.804684 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.804659 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-xbmhh"] Jul 14 14:39:24.843398 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.843361 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjkxr\" (UniqueName: \"kubernetes.io/projected/15bc9e76-9962-4bd8-a3e5-c00926e5b794-kube-api-access-kjkxr\") pod \"payload-processing-f9b8d9b9b-xbmhh\" (UID: \"15bc9e76-9962-4bd8-a3e5-c00926e5b794\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:24.843569 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.843411 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bc9e76-9962-4bd8-a3e5-c00926e5b794-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-xbmhh\" (UID: \"15bc9e76-9962-4bd8-a3e5-c00926e5b794\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:24.843569 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.843445 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lpl\" (UniqueName: \"kubernetes.io/projected/5b44233b-8db8-44b6-ad63-c360f042daf0-kube-api-access-t8lpl\") pod \"payload-pre-processing-74584bb9f8-sq8qt\" (UID: \"5b44233b-8db8-44b6-ad63-c360f042daf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:24.843569 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.843491 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b44233b-8db8-44b6-ad63-c360f042daf0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-sq8qt\" (UID: \"5b44233b-8db8-44b6-ad63-c360f042daf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:24.944729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.944683 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lpl\" (UniqueName: \"kubernetes.io/projected/5b44233b-8db8-44b6-ad63-c360f042daf0-kube-api-access-t8lpl\") pod \"payload-pre-processing-74584bb9f8-sq8qt\" (UID: \"5b44233b-8db8-44b6-ad63-c360f042daf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:24.944916 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.944742 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b44233b-8db8-44b6-ad63-c360f042daf0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-sq8qt\" (UID: \"5b44233b-8db8-44b6-ad63-c360f042daf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:24.944916 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.944791 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjkxr\" (UniqueName: \"kubernetes.io/projected/15bc9e76-9962-4bd8-a3e5-c00926e5b794-kube-api-access-kjkxr\") pod \"payload-processing-f9b8d9b9b-xbmhh\" (UID: \"15bc9e76-9962-4bd8-a3e5-c00926e5b794\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:24.944916 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.944827 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bc9e76-9962-4bd8-a3e5-c00926e5b794-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-xbmhh\" (UID: \"15bc9e76-9962-4bd8-a3e5-c00926e5b794\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:24.945537 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.945514 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b44233b-8db8-44b6-ad63-c360f042daf0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-sq8qt\" (UID: \"5b44233b-8db8-44b6-ad63-c360f042daf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:24.945604 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.945584 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bc9e76-9962-4bd8-a3e5-c00926e5b794-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-xbmhh\" (UID: \"15bc9e76-9962-4bd8-a3e5-c00926e5b794\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:24.953109 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.953083 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lpl\" (UniqueName: \"kubernetes.io/projected/5b44233b-8db8-44b6-ad63-c360f042daf0-kube-api-access-t8lpl\") pod \"payload-pre-processing-74584bb9f8-sq8qt\" (UID: \"5b44233b-8db8-44b6-ad63-c360f042daf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:24.953246 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:24.953119 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjkxr\" (UniqueName: \"kubernetes.io/projected/15bc9e76-9962-4bd8-a3e5-c00926e5b794-kube-api-access-kjkxr\") pod \"payload-processing-f9b8d9b9b-xbmhh\" (UID: \"15bc9e76-9962-4bd8-a3e5-c00926e5b794\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:25.072545 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:25.072444 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:25.102203 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:25.102166 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:25.245659 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:25.245635 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt"] Jul 14 14:39:25.247451 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:39:25.247424 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b44233b_8db8_44b6_ad63_c360f042daf0.slice/crio-934c33fea01af91774ca67ac5ef91d5a566fab4991950c98b0cc341ae81c6062 WatchSource:0}: Error finding container 934c33fea01af91774ca67ac5ef91d5a566fab4991950c98b0cc341ae81c6062: Status 404 returned error can't find the container with id 934c33fea01af91774ca67ac5ef91d5a566fab4991950c98b0cc341ae81c6062 Jul 14 14:39:25.268812 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:25.268788 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-xbmhh"] Jul 14 14:39:25.269719 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:39:25.269697 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bc9e76_9962_4bd8_a3e5_c00926e5b794.slice/crio-094ea3939f0d3095b4ff95149b39b3e08973fb9918dfbbeb80f0e8f20232427f WatchSource:0}: Error finding container 094ea3939f0d3095b4ff95149b39b3e08973fb9918dfbbeb80f0e8f20232427f: Status 404 returned error can't find the container with id 094ea3939f0d3095b4ff95149b39b3e08973fb9918dfbbeb80f0e8f20232427f Jul 14 14:39:25.516310 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:25.516274 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" event={"ID":"5b44233b-8db8-44b6-ad63-c360f042daf0","Type":"ContainerStarted","Data":"934c33fea01af91774ca67ac5ef91d5a566fab4991950c98b0cc341ae81c6062"} Jul 14 14:39:25.517556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:25.517528 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" event={"ID":"15bc9e76-9962-4bd8-a3e5-c00926e5b794","Type":"ContainerStarted","Data":"094ea3939f0d3095b4ff95149b39b3e08973fb9918dfbbeb80f0e8f20232427f"} Jul 14 14:39:27.531853 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:27.531810 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" event={"ID":"5b44233b-8db8-44b6-ad63-c360f042daf0","Type":"ContainerStarted","Data":"08ec959ba04990b8239f8f34bbe654a3e782734f0a2663d399dc02e08a376e13"} Jul 14 14:39:27.532405 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:27.531920 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:27.533365 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:27.533341 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" event={"ID":"15bc9e76-9962-4bd8-a3e5-c00926e5b794","Type":"ContainerStarted","Data":"bc7f6af462d9eee1dcf1111fd9b09c28d0f0e1eb6a6a6a24763b20137143dbfe"} Jul 14 14:39:27.533488 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:27.533404 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:27.561696 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:27.561646 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" podStartSLOduration=2.038683675 podStartE2EDuration="3.561630467s" totalImagesPullingTime="1.522946792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:27.559253791 +0000 UTC m=+556.994238415" watchObservedRunningTime="2026-07-14 14:39:27.561630467 +0000 UTC m=+556.996615090" Jul 14 14:39:27.590009 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:27.589941 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" podStartSLOduration=2.102215259 podStartE2EDuration="3.589923136s" totalImagesPullingTime="1.487707877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:27.589757154 +0000 UTC m=+557.024741779" watchObservedRunningTime="2026-07-14 14:39:27.589923136 +0000 UTC m=+557.024907759" Jul 14 14:39:32.149633 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:32.149388 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-wvnd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": read tcp 10.132.0.2:55636->10.132.0.38:15021: read: connection reset by peer" start-of-body= Jul 14 14:39:32.149633 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:32.149453 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" podUID="252f0ccf-34b0-4ab7-9641-c1b973022da1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": read tcp 10.132.0.2:55636->10.132.0.38:15021: read: connection reset by peer" Jul 14 14:39:32.559411 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:32.559379 2550 generic.go:356] "Generic (PLEG): container finished" podID="252f0ccf-34b0-4ab7-9641-c1b973022da1" containerID="4c23011a5fa332eccdd01963ce500ab7a80355a692b24e6dca36d98b5b21351d" exitCode=0 Jul 14 14:39:32.559590 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:32.559419 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" event={"ID":"252f0ccf-34b0-4ab7-9641-c1b973022da1","Type":"ContainerDied","Data":"4c23011a5fa332eccdd01963ce500ab7a80355a692b24e6dca36d98b5b21351d"} Jul 14 14:39:32.559801 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:32.559788 2550 scope.go:117] "RemoveContainer" containerID="4c23011a5fa332eccdd01963ce500ab7a80355a692b24e6dca36d98b5b21351d" Jul 14 14:39:32.560756 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:32.560728 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:39:32.560822 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:32.560806 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:39:32.560870 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:32.560855 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:39:33.566019 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:33.565983 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" event={"ID":"252f0ccf-34b0-4ab7-9641-c1b973022da1","Type":"ContainerStarted","Data":"b804a7ea2a9c1efc18adb9080baee26f3dd0a532e0f5f3856e4bad345ed79ae7"} Jul 14 14:39:34.546345 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:34.546298 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:39:34.546597 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:34.546570 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-wvnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 14 14:39:34.546679 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:34.546642 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" podUID="252f0ccf-34b0-4ab7-9641-c1b973022da1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 14 14:39:35.546452 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:35.546407 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-wvnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 14 14:39:35.546950 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:35.546478 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" podUID="252f0ccf-34b0-4ab7-9641-c1b973022da1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 14 14:39:36.546664 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:36.546624 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-wvnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 14 14:39:36.547090 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:36.546690 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" podUID="252f0ccf-34b0-4ab7-9641-c1b973022da1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 14 14:39:37.546065 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:37.546029 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-wvnd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 14 14:39:37.546254 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:37.546087 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" podUID="252f0ccf-34b0-4ab7-9641-c1b973022da1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 14 14:39:38.541039 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:38.541008 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-xbmhh" Jul 14 14:39:38.541501 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:38.541082 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-sq8qt" Jul 14 14:39:38.549146 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:38.549113 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:39:38.549415 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:38.549397 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:39:38.550179 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:38.550158 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-wvnd2" Jul 14 14:39:51.547904 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.547863 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7"] Jul 14 14:39:51.552072 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.552048 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.556098 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.556076 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 14:39:51.556098 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.556092 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 14 14:39:51.556268 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.556092 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tqhk5\"" Jul 14 14:39:51.556268 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.556130 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 14:39:51.571589 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.571560 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7"] Jul 14 14:39:51.678786 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.678722 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198fc64f-c361-40f7-92e7-090392167fa5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.678786 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.678795 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.679045 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.678812 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.679045 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.678850 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.679045 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.678934 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.679045 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.678989 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqz4\" (UniqueName: \"kubernetes.io/projected/198fc64f-c361-40f7-92e7-090392167fa5-kube-api-access-zcqz4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780125 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780074 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780125 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780128 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqz4\" (UniqueName: \"kubernetes.io/projected/198fc64f-c361-40f7-92e7-090392167fa5-kube-api-access-zcqz4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780388 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780159 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198fc64f-c361-40f7-92e7-090392167fa5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780388 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780192 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780388 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780207 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780388 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780345 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780645 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780619 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780769 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780653 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.780769 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.780672 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.782588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.782566 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198fc64f-c361-40f7-92e7-090392167fa5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.782723 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.782700 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198fc64f-c361-40f7-92e7-090392167fa5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.788915 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.788891 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqz4\" (UniqueName: \"kubernetes.io/projected/198fc64f-c361-40f7-92e7-090392167fa5-kube-api-access-zcqz4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zmxg7\" (UID: \"198fc64f-c361-40f7-92e7-090392167fa5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.862801 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.862707 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:39:51.999439 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:51.999409 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7"] Jul 14 14:39:52.001318 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:39:52.001289 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198fc64f_c361_40f7_92e7_090392167fa5.slice/crio-6f45dea2cd37c48d06c84dffb4d9e3495b6a12b7d98542e4320d4e28828d9970 WatchSource:0}: Error finding container 6f45dea2cd37c48d06c84dffb4d9e3495b6a12b7d98542e4320d4e28828d9970: Status 404 returned error can't find the container with id 6f45dea2cd37c48d06c84dffb4d9e3495b6a12b7d98542e4320d4e28828d9970 Jul 14 14:39:52.662255 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:52.662199 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" event={"ID":"198fc64f-c361-40f7-92e7-090392167fa5","Type":"ContainerStarted","Data":"6f45dea2cd37c48d06c84dffb4d9e3495b6a12b7d98542e4320d4e28828d9970"} Jul 14 14:39:53.669325 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:53.669281 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" event={"ID":"198fc64f-c361-40f7-92e7-090392167fa5","Type":"ContainerStarted","Data":"a6be337bb2e8496c30752b9bc5876f8ff138440e9aa1204ebfb649a40c2d8c15"} Jul 14 14:39:53.690922 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:39:53.690868 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" podStartSLOduration=1.510370175 podStartE2EDuration="2.690854477s" totalImagesPullingTime="1.180484302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:53.689272725 +0000 UTC m=+583.124257352" watchObservedRunningTime="2026-07-14 14:39:53.690854477 +0000 UTC m=+583.125839099" Jul 14 14:40:01.863603 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:01.863558 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:40:01.864215 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:01.863644 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:40:01.876123 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:01.876097 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:40:02.726853 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:02.726818 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zmxg7" Jul 14 14:40:05.548148 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.548114 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh"] Jul 14 14:40:05.551681 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.551661 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.555429 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.555406 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 14 14:40:05.565385 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.565357 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh"] Jul 14 14:40:05.606718 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.606682 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.606903 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.606728 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.606903 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.606749 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.606903 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.606770 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.606903 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.606784 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62386e8-230b-45e2-8160-246390b8746c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.606903 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.606802 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkc6n\" (UniqueName: \"kubernetes.io/projected/c62386e8-230b-45e2-8160-246390b8746c-kube-api-access-jkc6n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.707743 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.707695 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.707914 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.707750 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.707914 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.707786 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.707914 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.707808 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.707914 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.707825 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62386e8-230b-45e2-8160-246390b8746c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.707914 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.707850 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkc6n\" (UniqueName: \"kubernetes.io/projected/c62386e8-230b-45e2-8160-246390b8746c-kube-api-access-jkc6n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.708211 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.708187 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.708278 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.708228 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.708333 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.708308 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.710156 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.710135 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62386e8-230b-45e2-8160-246390b8746c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.710310 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.710293 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62386e8-230b-45e2-8160-246390b8746c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.716417 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.716391 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkc6n\" (UniqueName: \"kubernetes.io/projected/c62386e8-230b-45e2-8160-246390b8746c-kube-api-access-jkc6n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh\" (UID: \"c62386e8-230b-45e2-8160-246390b8746c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.862089 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.861997 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:05.997006 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:05.996979 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh"] Jul 14 14:40:06.000162 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:40:06.000136 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62386e8_230b_45e2_8160_246390b8746c.slice/crio-a2b4bb566e87c958e6c22a21c2a56e1d046398a11fe906affe9273d87966d698 WatchSource:0}: Error finding container a2b4bb566e87c958e6c22a21c2a56e1d046398a11fe906affe9273d87966d698: Status 404 returned error can't find the container with id a2b4bb566e87c958e6c22a21c2a56e1d046398a11fe906affe9273d87966d698 Jul 14 14:40:06.733082 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:06.732994 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" event={"ID":"c62386e8-230b-45e2-8160-246390b8746c","Type":"ContainerStarted","Data":"c35a3744e9ae87fa01a8d01fada3670ac5c3de0ad701fefb63868fd5a044eaa9"} Jul 14 14:40:06.733082 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:06.733052 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" event={"ID":"c62386e8-230b-45e2-8160-246390b8746c","Type":"ContainerStarted","Data":"a2b4bb566e87c958e6c22a21c2a56e1d046398a11fe906affe9273d87966d698"} Jul 14 14:40:06.752618 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:06.752563 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" podStartSLOduration=1.299941972 podStartE2EDuration="1.752547199s" totalImagesPullingTime="452.605227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:40:06.751412221 +0000 UTC m=+596.186396844" watchObservedRunningTime="2026-07-14 14:40:06.752547199 +0000 UTC m=+596.187531821" Jul 14 14:40:11.615424 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:11.615397 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:40:11.616117 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:11.616090 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:40:15.863052 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:15.863010 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:15.863541 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:15.863083 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:15.875431 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:15.875406 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:16.793109 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:16.793081 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh" Jul 14 14:40:17.250509 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.250429 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz"] Jul 14 14:40:17.254311 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.254290 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.257095 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.257070 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 14 14:40:17.263973 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.263930 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz"] Jul 14 14:40:17.318378 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.318325 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.318556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.318392 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.318556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.318459 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgt5\" (UniqueName: \"kubernetes.io/projected/7d21bbcb-60b4-4083-b6c9-87265f16814a-kube-api-access-6bgt5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.318556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.318496 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.318556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.318549 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.318816 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.318616 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d21bbcb-60b4-4083-b6c9-87265f16814a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.419807 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.419768 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.419807 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.419814 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgt5\" (UniqueName: \"kubernetes.io/projected/7d21bbcb-60b4-4083-b6c9-87265f16814a-kube-api-access-6bgt5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.420101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.419937 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.420172 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.420152 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.420228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.420208 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d21bbcb-60b4-4083-b6c9-87265f16814a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.420288 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.420254 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.420339 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.420285 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.420339 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.420321 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.420441 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.420424 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.422474 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.422448 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d21bbcb-60b4-4083-b6c9-87265f16814a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.422728 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.422712 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d21bbcb-60b4-4083-b6c9-87265f16814a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.427878 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.427861 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgt5\" (UniqueName: \"kubernetes.io/projected/7d21bbcb-60b4-4083-b6c9-87265f16814a-kube-api-access-6bgt5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz\" (UID: \"7d21bbcb-60b4-4083-b6c9-87265f16814a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.563796 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.563760 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:17.704125 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.704101 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz"] Jul 14 14:40:17.706472 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:40:17.706441 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d21bbcb_60b4_4083_b6c9_87265f16814a.slice/crio-468ce00eebbd32dea6da74ef4fb79697f8a313f03afa2bab339ca14a8c5521d1 WatchSource:0}: Error finding container 468ce00eebbd32dea6da74ef4fb79697f8a313f03afa2bab339ca14a8c5521d1: Status 404 returned error can't find the container with id 468ce00eebbd32dea6da74ef4fb79697f8a313f03afa2bab339ca14a8c5521d1 Jul 14 14:40:17.787697 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:17.787662 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" event={"ID":"7d21bbcb-60b4-4083-b6c9-87265f16814a","Type":"ContainerStarted","Data":"468ce00eebbd32dea6da74ef4fb79697f8a313f03afa2bab339ca14a8c5521d1"} Jul 14 14:40:18.793937 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:18.793897 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" event={"ID":"7d21bbcb-60b4-4083-b6c9-87265f16814a","Type":"ContainerStarted","Data":"714a338e386e0bd3f8c559262c1738aa6d2a98ca5d1b8cb1d35ee57cb74ecfa8"} Jul 14 14:40:18.811601 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:18.811549 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" podStartSLOduration=1.5814938280000002 podStartE2EDuration="1.811532826s" totalImagesPullingTime="230.038998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:40:18.810204188 +0000 UTC m=+608.245188825" watchObservedRunningTime="2026-07-14 14:40:18.811532826 +0000 UTC m=+608.246517448" Jul 14 14:40:27.564892 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:27.564842 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:27.564892 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:27.564899 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:27.577186 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:27.577140 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:40:27.850381 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:40:27.850287 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz" Jul 14 14:41:16.933375 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:16.933331 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f647cb5d6-b2fr2"] Jul 14 14:41:16.933979 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:16.933922 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" podUID="19b4ee44-4c9c-48f6-a24f-5ebee5d8f865" containerName="authorino" containerID="cri-o://0ca6368ca64975067d6dff714d32f4c485bfcf57277aa6723db83a355233bbcb" gracePeriod=30 Jul 14 14:41:17.081600 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.081564 2550 generic.go:356] "Generic (PLEG): container finished" podID="19b4ee44-4c9c-48f6-a24f-5ebee5d8f865" containerID="0ca6368ca64975067d6dff714d32f4c485bfcf57277aa6723db83a355233bbcb" exitCode=0 Jul 14 14:41:17.081758 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.081648 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" event={"ID":"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865","Type":"ContainerDied","Data":"0ca6368ca64975067d6dff714d32f4c485bfcf57277aa6723db83a355233bbcb"} Jul 14 14:41:17.186228 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.186166 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:41:17.270191 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.270153 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-tls-cert\") pod \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\" (UID: \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\") " Jul 14 14:41:17.270382 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.270218 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsvfr\" (UniqueName: \"kubernetes.io/projected/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-kube-api-access-lsvfr\") pod \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\" (UID: \"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865\") " Jul 14 14:41:17.272224 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.272191 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-kube-api-access-lsvfr" (OuterVolumeSpecName: "kube-api-access-lsvfr") pod "19b4ee44-4c9c-48f6-a24f-5ebee5d8f865" (UID: "19b4ee44-4c9c-48f6-a24f-5ebee5d8f865"). InnerVolumeSpecName "kube-api-access-lsvfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:41:17.282277 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.282253 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "19b4ee44-4c9c-48f6-a24f-5ebee5d8f865" (UID: "19b4ee44-4c9c-48f6-a24f-5ebee5d8f865"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 14:41:17.371686 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.371641 2550 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-tls-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:41:17.371686 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:17.371682 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsvfr\" (UniqueName: \"kubernetes.io/projected/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865-kube-api-access-lsvfr\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:41:18.088126 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:18.088089 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" event={"ID":"19b4ee44-4c9c-48f6-a24f-5ebee5d8f865","Type":"ContainerDied","Data":"9a4ae9028c8c3054b9e7bdab1d80189a47413b6dde957cd895cf9569bcc20ba5"} Jul 14 14:41:18.088637 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:18.088144 2550 scope.go:117] "RemoveContainer" containerID="0ca6368ca64975067d6dff714d32f4c485bfcf57277aa6723db83a355233bbcb" Jul 14 14:41:18.088637 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:18.088102 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f647cb5d6-b2fr2" Jul 14 14:41:18.114643 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:18.114610 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f647cb5d6-b2fr2"] Jul 14 14:41:18.121754 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:18.121719 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f647cb5d6-b2fr2"] Jul 14 14:41:19.120431 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:41:19.120402 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19b4ee44-4c9c-48f6-a24f-5ebee5d8f865" path="/var/lib/kubelet/pods/19b4ee44-4c9c-48f6-a24f-5ebee5d8f865/volumes" Jul 14 14:45:11.643805 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:45:11.643772 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:45:11.646127 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:45:11.646107 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:50:11.672641 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:50:11.672607 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:50:11.675126 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:50:11.674988 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:54:57.437242 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.437207 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n"] Jul 14 14:54:57.437740 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.437616 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b4ee44-4c9c-48f6-a24f-5ebee5d8f865" containerName="authorino" Jul 14 14:54:57.437740 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.437629 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b4ee44-4c9c-48f6-a24f-5ebee5d8f865" containerName="authorino" Jul 14 14:54:57.437740 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.437696 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="19b4ee44-4c9c-48f6-a24f-5ebee5d8f865" containerName="authorino" Jul 14 14:54:57.440604 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.440583 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.443196 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.443168 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2b2b03f7-openshift-default-dockercfg-52r4k\"" Jul 14 14:54:57.466870 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.466846 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n"] Jul 14 14:54:57.520040 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.519992 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-certs\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.520213 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.520073 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-envoy\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.520213 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.520095 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-data\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.520213 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.520141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istiod-ca-cert\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.520213 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.520201 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-token\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.520370 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.520252 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-credential-socket\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.520370 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.520301 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28zf\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-kube-api-access-s28zf\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.520370 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.520343 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-podinfo\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.520463 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.520381 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-socket\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.621739 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.621686 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-credential-socket\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.621739 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.621749 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28zf\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-kube-api-access-s28zf\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.621776 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-podinfo\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.621811 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-socket\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.621847 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-certs\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.621879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-envoy\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.621902 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-data\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622035 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.621957 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istiod-ca-cert\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622345 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.622061 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-token\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622345 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.622200 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-credential-socket\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622345 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.622274 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-certs\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622507 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.622381 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-socket\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622572 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.622550 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-data\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.622812 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.622794 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istiod-ca-cert\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.624148 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.624126 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-envoy\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.624385 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.624366 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-podinfo\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.630239 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.630213 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-token\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.630416 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.630400 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28zf\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-kube-api-access-s28zf\") pod \"e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.751653 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.751571 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:57.914220 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.914105 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n"] Jul 14 14:54:57.916661 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:54:57.916626 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a2cdda_d2ff_40a7_bebb_49e82877d7bb.slice/crio-1eba02f446b9bcfe93bd59b3fb3e908c0bcd2868fd23faa8f8e6c6f4ca06de3d WatchSource:0}: Error finding container 1eba02f446b9bcfe93bd59b3fb3e908c0bcd2868fd23faa8f8e6c6f4ca06de3d: Status 404 returned error can't find the container with id 1eba02f446b9bcfe93bd59b3fb3e908c0bcd2868fd23faa8f8e6c6f4ca06de3d Jul 14 14:54:57.918933 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.918894 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:54:57.919032 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.919011 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:54:57.919079 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:57.919063 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:54:58.082839 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:58.082800 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" event={"ID":"69a2cdda-d2ff-40a7-bebb-49e82877d7bb","Type":"ContainerStarted","Data":"65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417"} Jul 14 14:54:58.083049 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:58.082866 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" event={"ID":"69a2cdda-d2ff-40a7-bebb-49e82877d7bb","Type":"ContainerStarted","Data":"1eba02f446b9bcfe93bd59b3fb3e908c0bcd2868fd23faa8f8e6c6f4ca06de3d"} Jul 14 14:54:58.108146 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:58.108083 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" podStartSLOduration=1.108062921 podStartE2EDuration="1.108062921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:54:58.105527251 +0000 UTC m=+1487.540511876" watchObservedRunningTime="2026-07-14 14:54:58.108062921 +0000 UTC m=+1487.543047546" Jul 14 14:54:58.752110 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:58.752076 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:58.756987 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:58.756944 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:59.089042 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:59.089011 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:54:59.090109 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:54:59.090086 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:55:11.154643 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:11.154605 2550 generic.go:356] "Generic (PLEG): container finished" podID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerID="65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417" exitCode=0 Jul 14 14:55:11.155089 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:11.154678 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" event={"ID":"69a2cdda-d2ff-40a7-bebb-49e82877d7bb","Type":"ContainerDied","Data":"65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417"} Jul 14 14:55:11.155181 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:11.155162 2550 scope.go:117] "RemoveContainer" containerID="65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417" Jul 14 14:55:11.156187 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:11.156158 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:55:11.156248 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:11.156229 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:55:11.156284 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:11.156260 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:55:11.703118 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:11.703088 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:55:11.705536 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:11.705515 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 14:55:12.161642 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:12.161610 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" event={"ID":"69a2cdda-d2ff-40a7-bebb-49e82877d7bb","Type":"ContainerStarted","Data":"e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39"} Jul 14 14:55:12.752631 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:12.752594 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:55:12.753563 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:12.753538 2550 patch_prober.go:28] interesting pod/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 14 14:55:12.753674 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:12.753594 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 14 14:55:13.752735 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:13.752693 2550 patch_prober.go:28] interesting pod/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 14 14:55:13.753158 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:13.752753 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 14 14:55:14.089308 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:14.089267 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:55:14.752288 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:14.752245 2550 patch_prober.go:28] interesting pod/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 14 14:55:14.752496 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:14.752306 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 14 14:55:15.752504 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:15.752467 2550 patch_prober.go:28] interesting pod/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 14 14:55:15.752879 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:15.752520 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 14 14:55:16.756008 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:16.755974 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:55:16.756853 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:55:16.756833 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:56:28.993300 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:28.993264 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n"] Jul 14 14:56:28.993757 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:28.993498 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" containerID="cri-o://e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39" gracePeriod=30 Jul 14 14:56:29.555734 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.555707 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67"] Jul 14 14:56:29.559004 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.558983 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.561819 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.561801 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-11424108-gw-openshift-default-dockercfg-mmv5l\"" Jul 14 14:56:29.573554 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.573534 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67"] Jul 14 14:56:29.656776 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.656727 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-data\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.656982 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.656779 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74bcm\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-kube-api-access-74bcm\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.656982 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.656840 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-certs\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.656982 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.656928 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-socket\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.657189 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.656953 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istiod-ca-cert\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.657189 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.657017 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-podinfo\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.657189 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.657037 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-token\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.657189 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.657073 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-credential-socket\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.657189 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.657123 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-envoy\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.757764 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.757728 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-data\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.757925 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.757766 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74bcm\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-kube-api-access-74bcm\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.757925 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.757813 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-certs\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.757925 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.757859 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-socket\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.757925 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.757887 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istiod-ca-cert\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.757925 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.757914 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-podinfo\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.758225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.757944 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-token\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.758225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.758000 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-credential-socket\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.758225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.758031 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-envoy\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.758225 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.758192 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-data\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.758440 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.758249 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-certs\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.758440 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.758302 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-socket\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.758552 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.758458 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-credential-socket\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.758738 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.758715 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istiod-ca-cert\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.760322 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.760302 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-envoy\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.760588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.760567 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-podinfo\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.770182 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.770146 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-token\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.770444 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.770427 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74bcm\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-kube-api-access-74bcm\") pod \"e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:29.869027 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:29.868942 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:30.011199 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:30.011171 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67"] Jul 14 14:56:30.013516 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:56:30.013489 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fd9e4e_48e9_433b_92c8_3d7a7338b4c7.slice/crio-6489a03f78493954fd5fa73b908e507d4a552de53e31e993f258d4a60297c171 WatchSource:0}: Error finding container 6489a03f78493954fd5fa73b908e507d4a552de53e31e993f258d4a60297c171: Status 404 returned error can't find the container with id 6489a03f78493954fd5fa73b908e507d4a552de53e31e993f258d4a60297c171 Jul 14 14:56:30.015666 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:30.015634 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:56:30.015732 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:30.015701 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:56:30.015732 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:30.015727 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:56:30.548464 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:30.548427 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" event={"ID":"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7","Type":"ContainerStarted","Data":"1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28"} Jul 14 14:56:30.548661 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:30.548492 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" event={"ID":"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7","Type":"ContainerStarted","Data":"6489a03f78493954fd5fa73b908e507d4a552de53e31e993f258d4a60297c171"} Jul 14 14:56:30.572455 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:30.572401 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" podStartSLOduration=1.5723856619999999 podStartE2EDuration="1.572385662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:56:30.570406841 +0000 UTC m=+1580.005391463" watchObservedRunningTime="2026-07-14 14:56:30.572385662 +0000 UTC m=+1580.007370286" Jul 14 14:56:30.869679 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:30.869588 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:31.758969 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:31.758928 2550 patch_prober.go:28] interesting pod/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 14:56:31.759354 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:31.759028 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 14:56:31.874369 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:31.874340 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:32.558065 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:32.558031 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:32.558939 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:32.558918 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:34.245635 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.245608 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:56:34.299118 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299069 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s28zf\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-kube-api-access-s28zf\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.299396 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299145 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-token\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.299396 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299239 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-credential-socket\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.299396 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299271 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-envoy\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.299396 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299302 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-data\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.299396 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299347 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istiod-ca-cert\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.299396 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299373 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-podinfo\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.299714 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299405 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-socket\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.299714 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.299488 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-certs\") pod \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\" (UID: \"69a2cdda-d2ff-40a7-bebb-49e82877d7bb\") " Jul 14 14:56:34.300113 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.300085 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:56:34.301004 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.300423 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:56:34.301004 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.300934 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:56:34.301187 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.301138 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:56:34.301325 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.301298 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:56:34.302760 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.302729 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:56:34.302887 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.302814 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:56:34.303400 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.303375 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 14:56:34.304104 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.304084 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-kube-api-access-s28zf" (OuterVolumeSpecName: "kube-api-access-s28zf") pod "69a2cdda-d2ff-40a7-bebb-49e82877d7bb" (UID: "69a2cdda-d2ff-40a7-bebb-49e82877d7bb"). InnerVolumeSpecName "kube-api-access-s28zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:56:34.400568 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400525 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.400568 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400555 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s28zf\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-kube-api-access-s28zf\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.400568 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400566 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.400568 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400574 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.400824 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400583 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.400824 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400592 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.400824 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400600 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.400824 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400607 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.400824 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.400614 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a2cdda-d2ff-40a7-bebb-49e82877d7bb-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:56:34.570586 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.570497 2550 generic.go:356] "Generic (PLEG): container finished" podID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerID="e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39" exitCode=0 Jul 14 14:56:34.570586 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.570551 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" Jul 14 14:56:34.570764 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.570579 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" event={"ID":"69a2cdda-d2ff-40a7-bebb-49e82877d7bb","Type":"ContainerDied","Data":"e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39"} Jul 14 14:56:34.570764 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.570634 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n" event={"ID":"69a2cdda-d2ff-40a7-bebb-49e82877d7bb","Type":"ContainerDied","Data":"1eba02f446b9bcfe93bd59b3fb3e908c0bcd2868fd23faa8f8e6c6f4ca06de3d"} Jul 14 14:56:34.570764 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.570657 2550 scope.go:117] "RemoveContainer" containerID="e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39" Jul 14 14:56:34.580227 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.580209 2550 scope.go:117] "RemoveContainer" containerID="65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417" Jul 14 14:56:34.588235 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.588214 2550 scope.go:117] "RemoveContainer" containerID="e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39" Jul 14 14:56:34.588522 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:56:34.588497 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39\": container with ID starting with e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39 not found: ID does not exist" containerID="e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39" Jul 14 14:56:34.588635 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.588528 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39"} err="failed to get container status \"e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39\": rpc error: code = NotFound desc = could not find container \"e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39\": container with ID starting with e625f4aee369dc5b9e8a6c5a5f12a70e61b3a1bb6dafb295573a4f98fcc20a39 not found: ID does not exist" Jul 14 14:56:34.588635 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.588548 2550 scope.go:117] "RemoveContainer" containerID="65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417" Jul 14 14:56:34.588859 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:56:34.588843 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417\": container with ID starting with 65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417 not found: ID does not exist" containerID="65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417" Jul 14 14:56:34.588926 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.588863 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417"} err="failed to get container status \"65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417\": rpc error: code = NotFound desc = could not find container \"65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417\": container with ID starting with 65418c9a5e079f4d38a24fd5e14766209b8dd7b563163cd804ef190c43b25417 not found: ID does not exist" Jul 14 14:56:34.593714 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.593693 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n"] Jul 14 14:56:34.601684 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:34.601660 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2b2b03f7-openshift-default-6b4d59579f-xmd4n"] Jul 14 14:56:35.121249 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:35.121218 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" path="/var/lib/kubelet/pods/69a2cdda-d2ff-40a7-bebb-49e82877d7bb/volumes" Jul 14 14:56:47.559556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:47.559519 2550 patch_prober.go:28] interesting pod/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 14 14:56:47.560012 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:47.559577 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 14 14:56:47.645761 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:47.645731 2550 generic.go:356] "Generic (PLEG): container finished" podID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerID="1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28" exitCode=0 Jul 14 14:56:47.645956 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:47.645805 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" event={"ID":"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7","Type":"ContainerDied","Data":"1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28"} Jul 14 14:56:47.646247 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:47.646231 2550 scope.go:117] "RemoveContainer" containerID="1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28" Jul 14 14:56:47.647273 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:47.647243 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:56:47.647347 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:47.647332 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:56:47.647388 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:47.647369 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:56:48.653205 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:48.653168 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" event={"ID":"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7","Type":"ContainerStarted","Data":"b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c"} Jul 14 14:56:48.869696 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:48.869661 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:48.869875 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:48.869854 2550 patch_prober.go:28] interesting pod/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 14 14:56:48.869924 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:48.869901 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 14 14:56:49.869626 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:49.869590 2550 patch_prober.go:28] interesting pod/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 14 14:56:49.870048 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:49.869654 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 14 14:56:50.869700 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:50.869659 2550 patch_prober.go:28] interesting pod/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 14 14:56:50.870131 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:50.869733 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 14 14:56:51.870224 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:51.870187 2550 patch_prober.go:28] interesting pod/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 14 14:56:51.870623 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:51.870241 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 14 14:56:52.873140 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:52.873110 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:52.873519 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:52.873382 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:56:52.874136 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:56:52.874120 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:57:35.455931 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:35.455892 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67"] Jul 14 14:57:35.456518 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:35.456111 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" containerID="cri-o://b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c" gracePeriod=30 Jul 14 14:57:37.875744 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:37.875703 2550 patch_prober.go:28] interesting pod/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 14:57:37.876139 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:37.875763 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 14:57:40.718351 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.718326 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:57:40.898862 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.898821 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-envoy\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.899090 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.898873 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-podinfo\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.899090 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.898906 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-socket\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.899090 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.898922 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-token\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.899090 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.898973 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istiod-ca-cert\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.899090 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.899012 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-data\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.899372 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.899286 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-certs\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.899372 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.899330 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-credential-socket\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.899477 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.899421 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:57:40.899477 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.899437 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74bcm\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-kube-api-access-74bcm\") pod \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\" (UID: \"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7\") " Jul 14 14:57:40.900064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.899818 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:57:40.900064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.899843 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:57:40.900064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.899896 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:40.900064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.900029 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:57:40.900064 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.900037 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:57:40.901252 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.901227 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 14:57:40.901362 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.901345 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:57:40.901618 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.901592 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:57:40.901705 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.901612 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-kube-api-access-74bcm" (OuterVolumeSpecName: "kube-api-access-74bcm") pod "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" (UID: "62fd9e4e-48e9-433b-92c8-3d7a7338b4c7"). InnerVolumeSpecName "kube-api-access-74bcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:57:40.914642 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.914613 2550 generic.go:356] "Generic (PLEG): container finished" podID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerID="b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c" exitCode=0 Jul 14 14:57:40.914763 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.914655 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" Jul 14 14:57:40.914763 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.914691 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" event={"ID":"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7","Type":"ContainerDied","Data":"b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c"} Jul 14 14:57:40.914763 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.914758 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67" event={"ID":"62fd9e4e-48e9-433b-92c8-3d7a7338b4c7","Type":"ContainerDied","Data":"6489a03f78493954fd5fa73b908e507d4a552de53e31e993f258d4a60297c171"} Jul 14 14:57:40.914933 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.914790 2550 scope.go:117] "RemoveContainer" containerID="b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c" Jul 14 14:57:40.923533 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.923517 2550 scope.go:117] "RemoveContainer" containerID="1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28" Jul 14 14:57:40.931300 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.931277 2550 scope.go:117] "RemoveContainer" containerID="b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c" Jul 14 14:57:40.931536 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:57:40.931519 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c\": container with ID starting with b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c not found: ID does not exist" containerID="b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c" Jul 14 14:57:40.931599 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.931542 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c"} err="failed to get container status \"b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c\": rpc error: code = NotFound desc = could not find container \"b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c\": container with ID starting with b965fe0f9477a93c0f007a147e173f8d1bb1eb26e975fcdf9da8ee31f66a0b9c not found: ID does not exist" Jul 14 14:57:40.931599 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.931558 2550 scope.go:117] "RemoveContainer" containerID="1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28" Jul 14 14:57:40.931786 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:57:40.931771 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28\": container with ID starting with 1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28 not found: ID does not exist" containerID="1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28" Jul 14 14:57:40.931834 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.931790 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28"} err="failed to get container status \"1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28\": rpc error: code = NotFound desc = could not find container \"1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28\": container with ID starting with 1174d329320511fbdd537420d8114f1b843aaa36653b5f37ec83f9c1e5f8cc28 not found: ID does not exist" Jul 14 14:57:40.937177 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.937157 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67"] Jul 14 14:57:40.941739 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:40.941720 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-11424108-gw-openshift-default-6ffcb56d95-xlf67"] Jul 14 14:57:41.000570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.000539 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:41.000570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.000565 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:41.000570 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.000574 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:41.000791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.000585 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:41.000791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.000593 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:41.000791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.000600 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:41.000791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.000608 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:41.000791 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.000616 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74bcm\" (UniqueName: \"kubernetes.io/projected/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7-kube-api-access-74bcm\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:57:41.126671 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:41.126589 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" path="/var/lib/kubelet/pods/62fd9e4e-48e9-433b-92c8-3d7a7338b4c7/volumes" Jul 14 14:57:47.593276 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.593243 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd"] Jul 14 14:57:47.594268 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.594138 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" Jul 14 14:57:47.594268 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.594269 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" Jul 14 14:57:47.594456 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.594419 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" Jul 14 14:57:47.594456 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.594442 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" Jul 14 14:57:47.594560 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.594473 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" Jul 14 14:57:47.594560 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.594482 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" Jul 14 14:57:47.594748 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.594732 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" Jul 14 14:57:47.594806 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.594754 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" Jul 14 14:57:47.604330 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.604302 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.607326 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.607301 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9cfd4b9b-openshift-default-dockercfg-n7nnm\"" Jul 14 14:57:47.611671 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.611649 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd"] Jul 14 14:57:47.757082 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757040 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztchl\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-kube-api-access-ztchl\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.757082 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757077 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-token\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.757340 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757112 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-credential-socket\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.757340 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757219 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-certs\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.757340 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757268 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-data\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.757340 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757316 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-socket\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.757511 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757345 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8830e-e3af-46dc-8aaa-44fac9899e77-istiod-ca-cert\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.757511 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757382 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-envoy\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.757511 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.757407 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-podinfo\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.858603 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.858512 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-certs\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.858603 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.858561 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-data\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.858801 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.858668 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-socket\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.858801 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.858704 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8830e-e3af-46dc-8aaa-44fac9899e77-istiod-ca-cert\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.858908 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.858840 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-envoy\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.858908 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.858882 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-podinfo\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.858908 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.858892 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-certs\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.859101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.858943 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-data\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.859101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.859047 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-socket\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.859101 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.859051 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztchl\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-kube-api-access-ztchl\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.859245 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.859105 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-token\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.859245 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.859162 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-credential-socket\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.859447 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.859427 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8830e-e3af-46dc-8aaa-44fac9899e77-istiod-ca-cert\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.859505 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.859490 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-credential-socket\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.861281 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.861259 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-podinfo\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.861369 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.861263 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-envoy\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.871564 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.871534 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-token\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.871663 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.871647 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztchl\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-kube-api-access-ztchl\") pod \"e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:47.917484 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:47.917453 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:48.074145 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:48.074119 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd"] Jul 14 14:57:48.075736 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:57:48.075714 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c8830e_e3af_46dc_8aaa_44fac9899e77.slice/crio-f05369168ea046cecd9cd265d45b59a0e14fbf9efeb6539e22d8960c9830dfbb WatchSource:0}: Error finding container f05369168ea046cecd9cd265d45b59a0e14fbf9efeb6539e22d8960c9830dfbb: Status 404 returned error can't find the container with id f05369168ea046cecd9cd265d45b59a0e14fbf9efeb6539e22d8960c9830dfbb Jul 14 14:57:48.077781 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:48.077749 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:57:48.077838 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:48.077817 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:57:48.077881 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:48.077846 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:57:48.960917 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:48.960883 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" event={"ID":"21c8830e-e3af-46dc-8aaa-44fac9899e77","Type":"ContainerStarted","Data":"25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c"} Jul 14 14:57:48.961360 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:48.960942 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" event={"ID":"21c8830e-e3af-46dc-8aaa-44fac9899e77","Type":"ContainerStarted","Data":"f05369168ea046cecd9cd265d45b59a0e14fbf9efeb6539e22d8960c9830dfbb"} Jul 14 14:57:48.997166 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:48.997113 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" podStartSLOduration=1.997094922 podStartE2EDuration="1.997094922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:57:48.996524699 +0000 UTC m=+1658.431509326" watchObservedRunningTime="2026-07-14 14:57:48.997094922 +0000 UTC m=+1658.432079549" Jul 14 14:57:49.918219 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:49.918186 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:49.923284 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:49.923259 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:49.966690 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:49.966659 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:57:49.967465 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:57:49.967448 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:58:01.036400 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:01.036370 2550 generic.go:356] "Generic (PLEG): container finished" podID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerID="25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c" exitCode=0 Jul 14 14:58:01.036819 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:01.036452 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" event={"ID":"21c8830e-e3af-46dc-8aaa-44fac9899e77","Type":"ContainerDied","Data":"25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c"} Jul 14 14:58:01.036866 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:01.036854 2550 scope.go:117] "RemoveContainer" containerID="25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c" Jul 14 14:58:01.037990 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:01.037939 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:58:01.038082 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:01.038041 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:58:01.038123 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:01.038086 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:58:02.043213 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:02.043181 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" event={"ID":"21c8830e-e3af-46dc-8aaa-44fac9899e77","Type":"ContainerStarted","Data":"82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4"} Jul 14 14:58:02.919631 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:02.919583 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:58:02.919856 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:02.919730 2550 patch_prober.go:28] interesting pod/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 14:58:02.919856 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:02.919774 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 14:58:03.918761 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:03.918273 2550 patch_prober.go:28] interesting pod/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 14:58:03.918761 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:03.918349 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 14:58:04.918399 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:04.918350 2550 patch_prober.go:28] interesting pod/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 14:58:04.918617 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:04.918423 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 14:58:04.967069 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:04.967030 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:58:05.917871 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:05.917830 2550 patch_prober.go:28] interesting pod/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 14:58:05.918107 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:05.917901 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 14:58:06.922438 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:06.922406 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:58:06.923477 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:58:06.923455 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:59:09.941303 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:09.941264 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd"] Jul 14 14:59:09.941838 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:09.941531 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" containerID="cri-o://82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4" gracePeriod=30 Jul 14 14:59:10.302129 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.302095 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx"] Jul 14 14:59:10.303059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.302779 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" Jul 14 14:59:10.303059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.302800 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" Jul 14 14:59:10.303059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.302887 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a2cdda-d2ff-40a7-bebb-49e82877d7bb" containerName="istio-proxy" Jul 14 14:59:10.303059 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.302901 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fd9e4e-48e9-433b-92c8-3d7a7338b4c7" containerName="istio-proxy" Jul 14 14:59:10.306042 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.306024 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.308992 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.308946 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4b164646-openshift-default-dockercfg-hv5w2\"" Jul 14 14:59:10.321898 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.321871 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx"] Jul 14 14:59:10.387484 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387451 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-socket\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.387484 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387481 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-credential-socket\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.387717 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387518 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-token\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.387717 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387601 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istiod-ca-cert\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.387717 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387634 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-data\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.387717 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387659 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts8nl\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-kube-api-access-ts8nl\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.387717 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387708 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-podinfo\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.387931 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387744 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-envoy\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.387931 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.387767 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-certs\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.488537 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488493 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-podinfo\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.488729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488562 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-envoy\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.488729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488596 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-certs\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.488729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488623 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-socket\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.488729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488646 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-credential-socket\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.488729 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488693 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-token\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.489026 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488752 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istiod-ca-cert\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.489026 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488799 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-data\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.489026 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.488840 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8nl\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-kube-api-access-ts8nl\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.489185 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.489133 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-certs\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.489185 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.489142 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-socket\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.489434 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.489408 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-credential-socket\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.489545 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.489430 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-data\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.489588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.489554 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istiod-ca-cert\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.491111 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.491086 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-envoy\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.491212 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.491170 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-podinfo\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.498178 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.498155 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-token\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.498382 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.498363 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8nl\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-kube-api-access-ts8nl\") pod \"e2e-derive-4b164646-openshift-default-6647b85479-kczcx\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.616391 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.616306 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:10.758190 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.758163 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx"] Jul 14 14:59:10.759724 ip-10-0-141-194 kubenswrapper[2550]: W0714 14:59:10.759688 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72884cae_cb8f_44e2_81b8_6f0698d08fb1.slice/crio-a38c87025b6886968a29a8e2cb73ee138aca467d1351fd0bc2e66d54f27fcc82 WatchSource:0}: Error finding container a38c87025b6886968a29a8e2cb73ee138aca467d1351fd0bc2e66d54f27fcc82: Status 404 returned error can't find the container with id a38c87025b6886968a29a8e2cb73ee138aca467d1351fd0bc2e66d54f27fcc82 Jul 14 14:59:10.761891 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.761857 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:59:10.762003 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.761938 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:59:10.762072 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:10.761999 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:59:11.386017 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:11.385957 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" event={"ID":"72884cae-cb8f-44e2-81b8-6f0698d08fb1","Type":"ContainerStarted","Data":"691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9"} Jul 14 14:59:11.386412 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:11.386035 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" event={"ID":"72884cae-cb8f-44e2-81b8-6f0698d08fb1","Type":"ContainerStarted","Data":"a38c87025b6886968a29a8e2cb73ee138aca467d1351fd0bc2e66d54f27fcc82"} Jul 14 14:59:11.408726 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:11.408673 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" podStartSLOduration=1.408656493 podStartE2EDuration="1.408656493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:59:11.405923011 +0000 UTC m=+1740.840907634" watchObservedRunningTime="2026-07-14 14:59:11.408656493 +0000 UTC m=+1740.843641117" Jul 14 14:59:11.617079 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:11.617042 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:11.621789 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:11.621766 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:12.390843 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:12.390811 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:12.391639 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:12.391617 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:15.190398 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.190374 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:59:15.337134 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337087 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-certs\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337330 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337159 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-credential-socket\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337330 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337179 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-data\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337330 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337197 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8830e-e3af-46dc-8aaa-44fac9899e77-istiod-ca-cert\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337330 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337235 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-envoy\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337330 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337279 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-token\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337330 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337298 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztchl\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-kube-api-access-ztchl\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337679 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337391 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:59:15.337679 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337480 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-socket\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337679 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337513 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:59:15.337679 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337526 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-podinfo\") pod \"21c8830e-e3af-46dc-8aaa-44fac9899e77\" (UID: \"21c8830e-e3af-46dc-8aaa-44fac9899e77\") " Jul 14 14:59:15.337679 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337596 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c8830e-e3af-46dc-8aaa-44fac9899e77-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 14:59:15.337939 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337729 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:59:15.337939 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337895 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.337939 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337915 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.337939 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337931 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.338159 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.337944 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c8830e-e3af-46dc-8aaa-44fac9899e77-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.338466 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.338444 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-data" (OuterVolumeSpecName: "istio-data") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:59:15.339588 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.339561 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 14:59:15.339687 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.339582 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-kube-api-access-ztchl" (OuterVolumeSpecName: "kube-api-access-ztchl") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "kube-api-access-ztchl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:59:15.339813 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.339788 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-token" (OuterVolumeSpecName: "istio-token") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 14:59:15.339902 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.339823 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21c8830e-e3af-46dc-8aaa-44fac9899e77" (UID: "21c8830e-e3af-46dc-8aaa-44fac9899e77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 14:59:15.407193 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.407163 2550 generic.go:356] "Generic (PLEG): container finished" podID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerID="82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4" exitCode=0 Jul 14 14:59:15.407353 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.407214 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" Jul 14 14:59:15.407502 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.407240 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" event={"ID":"21c8830e-e3af-46dc-8aaa-44fac9899e77","Type":"ContainerDied","Data":"82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4"} Jul 14 14:59:15.407636 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.407616 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd" event={"ID":"21c8830e-e3af-46dc-8aaa-44fac9899e77","Type":"ContainerDied","Data":"f05369168ea046cecd9cd265d45b59a0e14fbf9efeb6539e22d8960c9830dfbb"} Jul 14 14:59:15.407696 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.407646 2550 scope.go:117] "RemoveContainer" containerID="82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4" Jul 14 14:59:15.423924 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.423904 2550 scope.go:117] "RemoveContainer" containerID="25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c" Jul 14 14:59:15.431623 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.431604 2550 scope.go:117] "RemoveContainer" containerID="82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4" Jul 14 14:59:15.431880 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:59:15.431863 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4\": container with ID starting with 82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4 not found: ID does not exist" containerID="82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4" Jul 14 14:59:15.431970 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.431887 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4"} err="failed to get container status \"82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4\": rpc error: code = NotFound desc = could not find container \"82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4\": container with ID starting with 82762f625a2ca33f8f43c51b1046680aef8ac23fed8e5e952e634a3b9ae62ea4 not found: ID does not exist" Jul 14 14:59:15.431970 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.431902 2550 scope.go:117] "RemoveContainer" containerID="25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c" Jul 14 14:59:15.432252 ip-10-0-141-194 kubenswrapper[2550]: E0714 14:59:15.432200 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c\": container with ID starting with 25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c not found: ID does not exist" containerID="25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c" Jul 14 14:59:15.432252 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.432241 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c"} err="failed to get container status \"25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c\": rpc error: code = NotFound desc = could not find container \"25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c\": container with ID starting with 25b415433c4a7eb4afddb8c08ac5aee073bca054f602cd1139681c597711be1c not found: ID does not exist" Jul 14 14:59:15.434587 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.434566 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd"] Jul 14 14:59:15.438479 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.438455 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.438479 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.438480 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.438646 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.438490 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.438646 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.438498 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztchl\" (UniqueName: \"kubernetes.io/projected/21c8830e-e3af-46dc-8aaa-44fac9899e77-kube-api-access-ztchl\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.438646 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.438507 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c8830e-e3af-46dc-8aaa-44fac9899e77-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 14:59:15.439474 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:15.439452 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9cfd4b9b-openshift-default-56cf79bbcc-tcwwd"] Jul 14 14:59:17.120556 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:17.120519 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" path="/var/lib/kubelet/pods/21c8830e-e3af-46dc-8aaa-44fac9899e77/volumes" Jul 14 14:59:28.051328 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:28.050985 2550 patch_prober.go:28] interesting pod/e2e-derive-4b164646-openshift-default-6647b85479-kczcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:34990->10.132.0.53:15021: read: connection reset by peer" start-of-body= Jul 14 14:59:28.051328 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:28.051073 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:34990->10.132.0.53:15021: read: connection reset by peer" Jul 14 14:59:28.472530 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:28.472432 2550 generic.go:356] "Generic (PLEG): container finished" podID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerID="691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9" exitCode=0 Jul 14 14:59:28.472530 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:28.472508 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" event={"ID":"72884cae-cb8f-44e2-81b8-6f0698d08fb1","Type":"ContainerDied","Data":"691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9"} Jul 14 14:59:28.473034 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:28.472994 2550 scope.go:117] "RemoveContainer" containerID="691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9" Jul 14 14:59:28.474004 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:28.473952 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:59:28.474095 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:28.474060 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:59:28.474157 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:28.474107 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 14:59:29.479423 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:29.479391 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" event={"ID":"72884cae-cb8f-44e2-81b8-6f0698d08fb1","Type":"ContainerStarted","Data":"2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2"} Jul 14 14:59:29.616858 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:29.616812 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:29.617344 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:29.617317 2550 patch_prober.go:28] interesting pod/e2e-derive-4b164646-openshift-default-6647b85479-kczcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 14 14:59:29.617567 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:29.617530 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 14 14:59:30.617621 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:30.617576 2550 patch_prober.go:28] interesting pod/e2e-derive-4b164646-openshift-default-6647b85479-kczcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 14 14:59:30.618152 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:30.617649 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 14 14:59:31.617321 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:31.617281 2550 patch_prober.go:28] interesting pod/e2e-derive-4b164646-openshift-default-6647b85479-kczcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 14 14:59:31.617532 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:31.617405 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 14 14:59:32.617057 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:32.617017 2550 patch_prober.go:28] interesting pod/e2e-derive-4b164646-openshift-default-6647b85479-kczcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 14 14:59:32.617707 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:32.617674 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 14 14:59:33.620083 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:33.620047 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:33.620509 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:33.620321 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 14:59:33.621116 ip-10-0-141-194 kubenswrapper[2550]: I0714 14:59:33.621090 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 15:00:11.731982 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:11.731942 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 15:00:11.735860 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:11.735841 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 15:00:31.072486 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:31.072449 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx"] Jul 14 15:00:31.072929 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:31.072682 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" containerID="cri-o://2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2" gracePeriod=30 Jul 14 15:00:33.622601 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:33.622564 2550 patch_prober.go:28] interesting pod/e2e-derive-4b164646-openshift-default-6647b85479-kczcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 15:00:33.623003 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:33.622635 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 15:00:36.330150 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.330118 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 15:00:36.488703 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.488659 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts8nl\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-kube-api-access-ts8nl\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.488858 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.488722 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-credential-socket\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.488858 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.488764 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-token\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.488858 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.488781 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-data\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.488858 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.488818 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-podinfo\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.489126 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.488863 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-socket\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.489126 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.488885 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-certs\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.489126 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.488913 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-envoy\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.489126 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.489008 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istiod-ca-cert\") pod \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\" (UID: \"72884cae-cb8f-44e2-81b8-6f0698d08fb1\") " Jul 14 15:00:36.489126 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.489087 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:00:36.489399 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.489222 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:00:36.489458 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.489403 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.489458 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.489425 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.489613 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.489572 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:00:36.489613 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.489587 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:00:36.489898 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.489874 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-data" (OuterVolumeSpecName: "istio-data") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:00:36.491160 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.491141 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-token" (OuterVolumeSpecName: "istio-token") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:00:36.491260 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.491156 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-kube-api-access-ts8nl" (OuterVolumeSpecName: "kube-api-access-ts8nl") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "kube-api-access-ts8nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:00:36.491412 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.491385 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:00:36.491456 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.491411 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72884cae-cb8f-44e2-81b8-6f0698d08fb1" (UID: "72884cae-cb8f-44e2-81b8-6f0698d08fb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:00:36.589865 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.589821 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts8nl\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-kube-api-access-ts8nl\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.589865 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.589857 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.589865 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.589870 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.590169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.589881 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.590169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.589893 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.590169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.589906 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.590169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.589917 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72884cae-cb8f-44e2-81b8-6f0698d08fb1-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:00:36.816770 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.816738 2550 generic.go:356] "Generic (PLEG): container finished" podID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerID="2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2" exitCode=0 Jul 14 15:00:36.816948 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.816784 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" Jul 14 15:00:36.816948 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.816829 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" event={"ID":"72884cae-cb8f-44e2-81b8-6f0698d08fb1","Type":"ContainerDied","Data":"2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2"} Jul 14 15:00:36.816948 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.816881 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx" event={"ID":"72884cae-cb8f-44e2-81b8-6f0698d08fb1","Type":"ContainerDied","Data":"a38c87025b6886968a29a8e2cb73ee138aca467d1351fd0bc2e66d54f27fcc82"} Jul 14 15:00:36.816948 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.816903 2550 scope.go:117] "RemoveContainer" containerID="2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2" Jul 14 15:00:36.826537 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.826521 2550 scope.go:117] "RemoveContainer" containerID="691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9" Jul 14 15:00:36.834677 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.834659 2550 scope.go:117] "RemoveContainer" containerID="2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2" Jul 14 15:00:36.835089 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:00:36.835066 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2\": container with ID starting with 2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2 not found: ID does not exist" containerID="2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2" Jul 14 15:00:36.835136 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.835099 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2"} err="failed to get container status \"2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2\": rpc error: code = NotFound desc = could not find container \"2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2\": container with ID starting with 2505abaa110adb38278fa173cad8ef112f3486bd2964d4ff14ee2c428e3ba7d2 not found: ID does not exist" Jul 14 15:00:36.835136 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.835117 2550 scope.go:117] "RemoveContainer" containerID="691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9" Jul 14 15:00:36.835383 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:00:36.835364 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9\": container with ID starting with 691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9 not found: ID does not exist" containerID="691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9" Jul 14 15:00:36.835423 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.835390 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9"} err="failed to get container status \"691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9\": rpc error: code = NotFound desc = could not find container \"691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9\": container with ID starting with 691c2acead61d8e1ec78efd32fb7d993dea69f91f3f53784cb69350bb3a13df9 not found: ID does not exist" Jul 14 15:00:36.841752 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.841727 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx"] Jul 14 15:00:36.846322 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:36.846296 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4b164646-openshift-default-6647b85479-kczcx"] Jul 14 15:00:37.120244 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:00:37.120167 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" path="/var/lib/kubelet/pods/72884cae-cb8f-44e2-81b8-6f0698d08fb1/volumes" Jul 14 15:02:07.900735 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.900695 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9"] Jul 14 15:02:07.901284 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901266 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" Jul 14 15:02:07.901335 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901288 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" Jul 14 15:02:07.901335 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901304 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" Jul 14 15:02:07.901335 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901312 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" Jul 14 15:02:07.901335 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901333 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" Jul 14 15:02:07.901468 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901343 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" Jul 14 15:02:07.901468 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901358 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" Jul 14 15:02:07.901468 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901366 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" Jul 14 15:02:07.901557 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901472 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" Jul 14 15:02:07.901557 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901485 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" Jul 14 15:02:07.901557 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901497 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="72884cae-cb8f-44e2-81b8-6f0698d08fb1" containerName="istio-proxy" Jul 14 15:02:07.901557 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.901508 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c8830e-e3af-46dc-8aaa-44fac9899e77" containerName="istio-proxy" Jul 14 15:02:07.903804 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.903786 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:07.907174 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.907156 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-beb2a668-openshift-default-dockercfg-ls8cj\"" Jul 14 15:02:07.937539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:07.937504 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9"] Jul 14 15:02:08.001806 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.001765 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lth\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-kube-api-access-r2lth\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.002018 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.001828 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-socket\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.002018 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.001871 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-certs\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.002018 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.001892 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-podinfo\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.002018 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.001914 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-envoy\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.002018 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.001931 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-token\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.002018 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.001974 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-data\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.002242 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.002062 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istiod-ca-cert\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.002242 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.002120 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-credential-socket\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.102866 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.102833 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lth\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-kube-api-access-r2lth\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103064 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.102890 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-socket\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103064 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.102919 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-certs\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103064 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.102938 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-podinfo\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103064 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.102978 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-envoy\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103064 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.102995 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-token\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103064 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.103020 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-data\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103064 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.103054 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istiod-ca-cert\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103429 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.103089 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-credential-socket\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103553 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.103417 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-certs\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103669 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.103639 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-socket\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.103903 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.103884 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istiod-ca-cert\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.104005 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.103910 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-data\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.104165 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.104143 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-credential-socket\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.105442 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.105418 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-podinfo\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.105608 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.105586 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-envoy\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.113651 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.113613 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lth\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-kube-api-access-r2lth\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.123123 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.123036 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-token\") pod \"e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.214294 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.214201 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:08.366054 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.366021 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9"] Jul 14 15:02:08.369750 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:02:08.369727 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f2ea16_2c27_4415_9bcc_a07b7d72e1f4.slice/crio-755c698b56cb65b8bfa58994f025b89d5f2697d05bb87ef4d7f3a6af04cd4cee WatchSource:0}: Error finding container 755c698b56cb65b8bfa58994f025b89d5f2697d05bb87ef4d7f3a6af04cd4cee: Status 404 returned error can't find the container with id 755c698b56cb65b8bfa58994f025b89d5f2697d05bb87ef4d7f3a6af04cd4cee Jul 14 15:02:08.371866 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.371827 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:02:08.371951 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.371899 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:02:08.371951 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:08.371926 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:02:09.266212 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:09.266173 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" event={"ID":"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4","Type":"ContainerStarted","Data":"91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7"} Jul 14 15:02:09.266599 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:09.266238 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" event={"ID":"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4","Type":"ContainerStarted","Data":"755c698b56cb65b8bfa58994f025b89d5f2697d05bb87ef4d7f3a6af04cd4cee"} Jul 14 15:02:09.303521 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:09.303468 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" podStartSLOduration=2.303453009 podStartE2EDuration="2.303453009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:02:09.301518182 +0000 UTC m=+1918.736502809" watchObservedRunningTime="2026-07-14 15:02:09.303453009 +0000 UTC m=+1918.738437632" Jul 14 15:02:10.214851 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:10.214812 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:10.219814 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:10.219788 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:10.271556 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:10.271526 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:10.272577 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:10.272557 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:24.351663 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:24.351573 2550 generic.go:356] "Generic (PLEG): container finished" podID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerID="91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7" exitCode=0 Jul 14 15:02:24.351663 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:24.351648 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" event={"ID":"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4","Type":"ContainerDied","Data":"91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7"} Jul 14 15:02:24.352209 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:24.352071 2550 scope.go:117] "RemoveContainer" containerID="91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7" Jul 14 15:02:24.353156 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:24.353123 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:02:24.353232 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:24.353211 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:02:24.353274 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:24.353264 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:02:25.271463 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:25.271422 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:25.358615 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:25.358578 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" event={"ID":"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4","Type":"ContainerStarted","Data":"59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701"} Jul 14 15:02:26.215981 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:26.215917 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:26.216199 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:26.216131 2550 patch_prober.go:28] interesting pod/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 15:02:26.216281 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:26.216191 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 15:02:27.215534 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:27.215490 2550 patch_prober.go:28] interesting pod/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 15:02:27.216113 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:27.215565 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 15:02:28.215276 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:28.214741 2550 patch_prober.go:28] interesting pod/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 15:02:28.215276 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:28.214855 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 15:02:29.214727 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:29.214689 2550 patch_prober.go:28] interesting pod/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 15:02:29.215255 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:29.214777 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 15:02:30.219105 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:30.219072 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:30.219547 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:30.219392 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:30.220128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:30.220112 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:48.679724 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:48.679691 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9"] Jul 14 15:02:48.680269 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:48.679916 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" containerID="cri-o://59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701" gracePeriod=30 Jul 14 15:02:53.939817 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:53.939789 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:54.061770 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.061731 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-token\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.061770 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.061776 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-envoy\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.062169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.061797 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-data\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.062169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.061831 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-podinfo\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.062169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.061869 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istiod-ca-cert\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.062169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.061930 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-credential-socket\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.062169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.061986 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2lth\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-kube-api-access-r2lth\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.062169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062038 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-socket\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.062169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062099 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-certs\") pod \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\" (UID: \"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4\") " Jul 14 15:02:54.062523 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062277 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:02:54.062523 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062326 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:02:54.062523 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062468 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:02:54.062523 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062489 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:02:54.062722 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062625 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.062722 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062645 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.062722 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062660 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.062722 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062673 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.062996 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.062951 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:02:54.064238 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.064209 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:02:54.064347 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.064284 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:02:54.064474 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.064458 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:02:54.064533 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.064471 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-kube-api-access-r2lth" (OuterVolumeSpecName: "kube-api-access-r2lth") pod "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" (UID: "55f2ea16-2c27-4415-9bcc-a07b7d72e1f4"). InnerVolumeSpecName "kube-api-access-r2lth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:02:54.163862 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.163814 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.163862 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.163856 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.163862 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.163865 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.163862 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.163873 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.163862 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.163882 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2lth\" (UniqueName: \"kubernetes.io/projected/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4-kube-api-access-r2lth\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:02:54.503264 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.503170 2550 generic.go:356] "Generic (PLEG): container finished" podID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerID="59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701" exitCode=0 Jul 14 15:02:54.503264 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.503221 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" Jul 14 15:02:54.503264 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.503223 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" event={"ID":"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4","Type":"ContainerDied","Data":"59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701"} Jul 14 15:02:54.503549 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.503297 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9" event={"ID":"55f2ea16-2c27-4415-9bcc-a07b7d72e1f4","Type":"ContainerDied","Data":"755c698b56cb65b8bfa58994f025b89d5f2697d05bb87ef4d7f3a6af04cd4cee"} Jul 14 15:02:54.503549 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.503326 2550 scope.go:117] "RemoveContainer" containerID="59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701" Jul 14 15:02:54.512864 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.512847 2550 scope.go:117] "RemoveContainer" containerID="91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7" Jul 14 15:02:54.521137 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.521115 2550 scope.go:117] "RemoveContainer" containerID="59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701" Jul 14 15:02:54.521412 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:02:54.521393 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701\": container with ID starting with 59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701 not found: ID does not exist" containerID="59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701" Jul 14 15:02:54.521469 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.521424 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701"} err="failed to get container status \"59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701\": rpc error: code = NotFound desc = could not find container \"59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701\": container with ID starting with 59726c86f6dba2816ce5ac43ee17c19279b6f7c194158eb7b21c9511a82b8701 not found: ID does not exist" Jul 14 15:02:54.521469 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.521444 2550 scope.go:117] "RemoveContainer" containerID="91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7" Jul 14 15:02:54.521672 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:02:54.521654 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7\": container with ID starting with 91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7 not found: ID does not exist" containerID="91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7" Jul 14 15:02:54.521731 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.521682 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7"} err="failed to get container status \"91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7\": rpc error: code = NotFound desc = could not find container \"91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7\": container with ID starting with 91ba5d2b427acbb65eaa13d31635610f0d2bc86b3aca0d7c3ba8efb7221d4eb7 not found: ID does not exist" Jul 14 15:02:54.529449 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.529411 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9"] Jul 14 15:02:54.539278 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:54.539251 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-beb2a668-openshift-default-5b7dc648d7-zl4r9"] Jul 14 15:02:55.120538 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:02:55.120507 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" path="/var/lib/kubelet/pods/55f2ea16-2c27-4415-9bcc-a07b7d72e1f4/volumes" Jul 14 15:03:08.475605 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.475573 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l"] Jul 14 15:03:08.475998 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.475931 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" Jul 14 15:03:08.475998 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.475942 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" Jul 14 15:03:08.476077 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.476025 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" Jul 14 15:03:08.476077 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.476035 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" Jul 14 15:03:08.480622 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.480600 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.484063 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.484045 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3fe6f3d4-openshift-default-dockercfg-v75sk\"" Jul 14 15:03:08.491938 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.491913 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-envoy\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.492049 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.491946 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-socket\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.492049 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.492012 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-data\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.492049 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.492039 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-token\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.492167 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.492100 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2ht\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-kube-api-access-wr2ht\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.492167 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.492127 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-podinfo\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.492167 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.492159 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istiod-ca-cert\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.492286 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.492219 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-credential-socket\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.492286 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.492266 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-certs\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.508561 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.508527 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l"] Jul 14 15:03:08.593311 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593274 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-token\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593518 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593327 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2ht\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-kube-api-access-wr2ht\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593518 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593347 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-podinfo\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593518 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593366 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istiod-ca-cert\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593518 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593410 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-credential-socket\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593518 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593457 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-certs\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593518 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593493 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-envoy\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593518 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593517 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-socket\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593882 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593575 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-data\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.593939 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.593904 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-credential-socket\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.594247 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.594221 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-certs\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.594330 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.594259 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-socket\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.594390 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.594325 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istiod-ca-cert\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.594390 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.594355 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-data\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.595909 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.595885 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-envoy\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.596228 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.596207 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-podinfo\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.604312 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.603913 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-token\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.604312 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.603922 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2ht\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-kube-api-access-wr2ht\") pod \"e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.793124 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.793088 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:08.953179 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:03:08.953142 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf211ddee_7390_4a02_b2fb_cf61ad84ea45.slice/crio-2c68ff11b43e9f0af0139f91568c03c4aeed39fb1704036616b95f7f84682875 WatchSource:0}: Error finding container 2c68ff11b43e9f0af0139f91568c03c4aeed39fb1704036616b95f7f84682875: Status 404 returned error can't find the container with id 2c68ff11b43e9f0af0139f91568c03c4aeed39fb1704036616b95f7f84682875 Jul 14 15:03:08.954915 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.954869 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l"] Jul 14 15:03:08.955450 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.955417 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:03:08.955504 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.955486 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:03:08.955550 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:08.955511 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:03:09.587317 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:09.587286 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" event={"ID":"f211ddee-7390-4a02-b2fb-cf61ad84ea45","Type":"ContainerStarted","Data":"af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd"} Jul 14 15:03:09.587702 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:09.587339 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" event={"ID":"f211ddee-7390-4a02-b2fb-cf61ad84ea45","Type":"ContainerStarted","Data":"2c68ff11b43e9f0af0139f91568c03c4aeed39fb1704036616b95f7f84682875"} Jul 14 15:03:09.630241 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:09.630186 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" podStartSLOduration=1.630170994 podStartE2EDuration="1.630170994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:03:09.627467464 +0000 UTC m=+1979.062452087" watchObservedRunningTime="2026-07-14 15:03:09.630170994 +0000 UTC m=+1979.065155618" Jul 14 15:03:09.793205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:09.793153 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:10.799996 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:10.799950 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:11.597648 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:11.597607 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:11.598525 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:11.598502 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:25.667473 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:25.667438 2550 generic.go:356] "Generic (PLEG): container finished" podID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerID="af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd" exitCode=0 Jul 14 15:03:25.667887 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:25.667515 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" event={"ID":"f211ddee-7390-4a02-b2fb-cf61ad84ea45","Type":"ContainerDied","Data":"af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd"} Jul 14 15:03:25.667930 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:25.667909 2550 scope.go:117] "RemoveContainer" containerID="af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd" Jul 14 15:03:25.668899 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:25.668872 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:03:25.669002 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:25.668983 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:03:25.669048 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:25.669032 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:03:26.598412 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:26.598374 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:26.675633 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:26.675601 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" event={"ID":"f211ddee-7390-4a02-b2fb-cf61ad84ea45","Type":"ContainerStarted","Data":"40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64"} Jul 14 15:03:26.793694 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:26.793660 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:26.794656 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:26.794637 2550 patch_prober.go:28] interesting pod/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 15:03:26.794720 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:26.794690 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 15:03:27.793820 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:27.793783 2550 patch_prober.go:28] interesting pod/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 15:03:27.794233 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:27.793839 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 15:03:28.793684 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:28.793636 2550 patch_prober.go:28] interesting pod/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 15:03:28.793892 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:28.793730 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 15:03:29.794300 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:29.794268 2550 patch_prober.go:28] interesting pod/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 15:03:29.794848 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:29.794319 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 15:03:30.794277 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:30.794241 2550 patch_prober.go:28] interesting pod/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 14 15:03:30.794455 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:30.794297 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 14 15:03:31.797744 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:31.797710 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:31.798205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:31.798070 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:03:31.798777 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:03:31.798758 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:05:06.738004 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:06.737906 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l"] Jul 14 15:05:06.738634 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:06.738206 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" containerID="cri-o://40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64" gracePeriod=30 Jul 14 15:05:11.761652 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:11.761626 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 15:05:11.767478 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:11.767458 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 15:05:12.002605 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.002542 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:05:12.125997 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.125915 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istiod-ca-cert\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.125997 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.125984 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-socket\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.125997 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126004 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-data\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.126312 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126101 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-token\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.126312 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126134 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-podinfo\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.126312 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126161 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-envoy\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.126312 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126182 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-certs\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.126312 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126254 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-credential-socket\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.126312 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126278 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:05:12.126630 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126339 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr2ht\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-kube-api-access-wr2ht\") pod \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\" (UID: \"f211ddee-7390-4a02-b2fb-cf61ad84ea45\") " Jul 14 15:05:12.126630 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126510 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:05:12.126630 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126538 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:05:12.126818 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126689 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.126818 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126706 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.126818 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126720 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.126995 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126833 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-data" (OuterVolumeSpecName: "istio-data") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:05:12.126995 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.126906 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:05:12.128442 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.128415 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-token" (OuterVolumeSpecName: "istio-token") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:05:12.128572 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.128492 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-kube-api-access-wr2ht" (OuterVolumeSpecName: "kube-api-access-wr2ht") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "kube-api-access-wr2ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:05:12.128814 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.128784 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:05:12.128908 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.128868 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f211ddee-7390-4a02-b2fb-cf61ad84ea45" (UID: "f211ddee-7390-4a02-b2fb-cf61ad84ea45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:05:12.217047 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.217014 2550 generic.go:356] "Generic (PLEG): container finished" podID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerID="40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64" exitCode=0 Jul 14 15:05:12.217218 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.217055 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" event={"ID":"f211ddee-7390-4a02-b2fb-cf61ad84ea45","Type":"ContainerDied","Data":"40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64"} Jul 14 15:05:12.217218 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.217063 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" Jul 14 15:05:12.217218 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.217103 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l" event={"ID":"f211ddee-7390-4a02-b2fb-cf61ad84ea45","Type":"ContainerDied","Data":"2c68ff11b43e9f0af0139f91568c03c4aeed39fb1704036616b95f7f84682875"} Jul 14 15:05:12.217218 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.217125 2550 scope.go:117] "RemoveContainer" containerID="40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64" Jul 14 15:05:12.229895 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.229585 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2ht\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-kube-api-access-wr2ht\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.229895 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.229614 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.229895 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.229630 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.229895 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.229643 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.229895 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.229656 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.229895 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.229669 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f211ddee-7390-4a02-b2fb-cf61ad84ea45-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:05:12.230611 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.230591 2550 scope.go:117] "RemoveContainer" containerID="af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd" Jul 14 15:05:12.238987 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.238953 2550 scope.go:117] "RemoveContainer" containerID="40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64" Jul 14 15:05:12.239308 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:05:12.239284 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64\": container with ID starting with 40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64 not found: ID does not exist" containerID="40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64" Jul 14 15:05:12.239433 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.239344 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64"} err="failed to get container status \"40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64\": rpc error: code = NotFound desc = could not find container \"40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64\": container with ID starting with 40da23ddc521fe007d43d415d5acf7347fe212b12b7529a3a51e3216c049bb64 not found: ID does not exist" Jul 14 15:05:12.239433 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.239370 2550 scope.go:117] "RemoveContainer" containerID="af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd" Jul 14 15:05:12.239752 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:05:12.239734 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd\": container with ID starting with af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd not found: ID does not exist" containerID="af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd" Jul 14 15:05:12.239818 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.239757 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd"} err="failed to get container status \"af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd\": rpc error: code = NotFound desc = could not find container \"af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd\": container with ID starting with af0f489c49745ed02712b02267d94a3d8dab71b472ca80f1082d32bc21d67efd not found: ID does not exist" Jul 14 15:05:12.241508 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.241484 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l"] Jul 14 15:05:12.245198 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:12.245174 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3fe6f3d4-openshift-default-58ffc768bc-82s5l"] Jul 14 15:05:13.120508 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:05:13.120476 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" path="/var/lib/kubelet/pods/f211ddee-7390-4a02-b2fb-cf61ad84ea45/volumes" Jul 14 15:06:01.827673 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.827634 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n"] Jul 14 15:06:01.828205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.828066 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" Jul 14 15:06:01.828205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.828079 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" Jul 14 15:06:01.828205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.828087 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" Jul 14 15:06:01.828205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.828094 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" Jul 14 15:06:01.828205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.828105 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" Jul 14 15:06:01.828205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.828113 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f2ea16-2c27-4415-9bcc-a07b7d72e1f4" containerName="istio-proxy" Jul 14 15:06:01.828205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.828180 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" Jul 14 15:06:01.828205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.828188 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f211ddee-7390-4a02-b2fb-cf61ad84ea45" containerName="istio-proxy" Jul 14 15:06:01.831058 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.831042 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.833808 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.833785 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-03dca1-openshift-default-dockercfg-fjhk6\"" Jul 14 15:06:01.842916 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.842888 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n"] Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979295 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-envoy\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979375 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istiod-ca-cert\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979416 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-credential-socket\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979461 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-token\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979516 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-socket\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979550 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-podinfo\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979633 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-certs\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979666 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-data\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:01.980994 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:01.979709 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rtv\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-kube-api-access-r7rtv\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080605 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080503 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-envoy\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080605 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080556 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istiod-ca-cert\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080605 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080583 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-credential-socket\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080901 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080613 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-token\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080901 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080654 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-socket\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080901 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080682 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-podinfo\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080901 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080740 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-certs\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080901 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080762 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-data\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.080901 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.080797 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rtv\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-kube-api-access-r7rtv\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.081258 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.081126 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-credential-socket\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.081319 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.081276 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-certs\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.081381 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.081331 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-data\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.081437 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.081395 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-socket\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.081437 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.081423 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istiod-ca-cert\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.082896 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.082879 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-envoy\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.083487 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.083467 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-podinfo\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.091154 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.091116 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-token\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.091818 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.091792 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rtv\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-kube-api-access-r7rtv\") pod \"e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.142139 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.142103 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:02.298721 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.298693 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n"] Jul 14 15:06:02.301361 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:06:02.301333 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce34ba4_7a22_416f_aa50_898f62c2d83a.slice/crio-fd87f5f506c97108224d1bdebfc10ee81307d3f6ca469ef92a17c13e06c2ced0 WatchSource:0}: Error finding container fd87f5f506c97108224d1bdebfc10ee81307d3f6ca469ef92a17c13e06c2ced0: Status 404 returned error can't find the container with id fd87f5f506c97108224d1bdebfc10ee81307d3f6ca469ef92a17c13e06c2ced0 Jul 14 15:06:02.303546 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.303510 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:02.303662 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.303575 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:02.303662 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.303603 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:02.459665 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.459622 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" event={"ID":"4ce34ba4-7a22-416f-aa50-898f62c2d83a","Type":"ContainerStarted","Data":"62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6"} Jul 14 15:06:02.459820 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.459693 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" event={"ID":"4ce34ba4-7a22-416f-aa50-898f62c2d83a","Type":"ContainerStarted","Data":"fd87f5f506c97108224d1bdebfc10ee81307d3f6ca469ef92a17c13e06c2ced0"} Jul 14 15:06:02.484177 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:02.484113 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" podStartSLOduration=1.484093983 podStartE2EDuration="1.484093983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:06:02.483491107 +0000 UTC m=+2151.918475731" watchObservedRunningTime="2026-07-14 15:06:02.484093983 +0000 UTC m=+2151.919078609" Jul 14 15:06:03.142368 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:03.142272 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:03.147176 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:03.147147 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:03.464846 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:03.464755 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:03.465864 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:03.465841 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:09.586762 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.586728 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml"] Jul 14 15:06:09.590573 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.590553 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.593904 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.593881 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4226a2-openshift-default-dockercfg-9tzqd\"" Jul 14 15:06:09.608241 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.608208 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml"] Jul 14 15:06:09.756080 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756042 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-token\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.756256 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756094 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9ncj\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-kube-api-access-m9ncj\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.756256 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756178 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-certs\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.756256 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756247 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-credential-socket\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.756383 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756305 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-socket\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.756383 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756331 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-podinfo\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.756383 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756353 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istiod-ca-cert\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.756504 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756389 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-data\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.756504 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.756418 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-envoy\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857553 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857448 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-credential-socket\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857553 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857526 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-socket\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857553 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857551 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-podinfo\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857859 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857579 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istiod-ca-cert\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857859 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857615 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-data\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857859 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857655 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-envoy\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857859 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857739 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-token\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857859 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857778 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9ncj\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-kube-api-access-m9ncj\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.857859 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857811 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-certs\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.858217 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.857920 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-credential-socket\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.858217 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.858199 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-certs\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.858329 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.858308 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-data\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.858444 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.858414 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istiod-ca-cert\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.858570 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.858476 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-socket\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.860390 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.860365 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-podinfo\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.860390 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.860383 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-envoy\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.866907 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.866876 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-token\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.867035 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.867000 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9ncj\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-kube-api-access-m9ncj\") pod \"e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:09.903045 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:09.903010 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:10.053818 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:10.053743 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml"] Jul 14 15:06:10.056700 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:06:10.056668 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7154aea7_a868_43ab_acd5_ef45ad0ea6f0.slice/crio-9171add066cb89eec475f4aa57b1202a2d9118ceaa44ac12afd77ef55f9fad12 WatchSource:0}: Error finding container 9171add066cb89eec475f4aa57b1202a2d9118ceaa44ac12afd77ef55f9fad12: Status 404 returned error can't find the container with id 9171add066cb89eec475f4aa57b1202a2d9118ceaa44ac12afd77ef55f9fad12 Jul 14 15:06:10.058924 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:10.058894 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:10.059017 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:10.058974 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:10.059017 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:10.059006 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:10.502267 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:10.502183 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" event={"ID":"7154aea7-a868-43ab-acd5-ef45ad0ea6f0","Type":"ContainerStarted","Data":"c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74"} Jul 14 15:06:10.502267 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:10.502243 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" event={"ID":"7154aea7-a868-43ab-acd5-ef45ad0ea6f0","Type":"ContainerStarted","Data":"9171add066cb89eec475f4aa57b1202a2d9118ceaa44ac12afd77ef55f9fad12"} Jul 14 15:06:10.527076 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:10.527016 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" podStartSLOduration=1.52699498 podStartE2EDuration="1.52699498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:06:10.524862095 +0000 UTC m=+2159.959846732" watchObservedRunningTime="2026-07-14 15:06:10.52699498 +0000 UTC m=+2159.961979600" Jul 14 15:06:10.904151 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:10.904104 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:11.908547 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:11.908518 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:12.513606 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:12.513573 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:12.514616 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:12.514594 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:19.507123 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:19.507079 2550 patch_prober.go:28] interesting pod/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 15:06:19.507495 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:19.507139 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 15:06:20.557610 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:20.557578 2550 generic.go:356] "Generic (PLEG): container finished" podID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerID="62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6" exitCode=0 Jul 14 15:06:20.558128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:20.557665 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" event={"ID":"4ce34ba4-7a22-416f-aa50-898f62c2d83a","Type":"ContainerDied","Data":"62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6"} Jul 14 15:06:20.558128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:20.558116 2550 scope.go:117] "RemoveContainer" containerID="62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6" Jul 14 15:06:20.559162 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:20.559136 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:20.559222 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:20.559211 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:20.559263 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:20.559243 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:21.564652 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:21.564616 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" event={"ID":"4ce34ba4-7a22-416f-aa50-898f62c2d83a","Type":"ContainerStarted","Data":"353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f"} Jul 14 15:06:22.143155 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:22.143108 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:22.143461 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:22.143170 2550 patch_prober.go:28] interesting pod/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 14 15:06:22.143633 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:22.143610 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 14 15:06:23.143489 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:23.143447 2550 patch_prober.go:28] interesting pod/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 14 15:06:23.144133 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:23.144098 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 14 15:06:24.142754 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:24.142714 2550 patch_prober.go:28] interesting pod/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 14 15:06:24.142931 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:24.142773 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 14 15:06:25.143531 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:25.143494 2550 patch_prober.go:28] interesting pod/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 14 15:06:25.143918 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:25.143549 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 14 15:06:26.146347 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:26.146305 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:26.146802 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:26.146784 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:26.147434 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:26.147416 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:06:27.515144 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:27.515106 2550 patch_prober.go:28] interesting pod/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 15:06:27.515561 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:27.515162 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 15:06:27.597354 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:27.597320 2550 generic.go:356] "Generic (PLEG): container finished" podID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerID="c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74" exitCode=0 Jul 14 15:06:27.597528 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:27.597392 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" event={"ID":"7154aea7-a868-43ab-acd5-ef45ad0ea6f0","Type":"ContainerDied","Data":"c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74"} Jul 14 15:06:27.597906 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:27.597888 2550 scope.go:117] "RemoveContainer" containerID="c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74" Jul 14 15:06:27.598931 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:27.598904 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:27.599044 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:27.598985 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:27.599044 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:27.599017 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:06:28.607779 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:28.607732 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" event={"ID":"7154aea7-a868-43ab-acd5-ef45ad0ea6f0","Type":"ContainerStarted","Data":"b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f"} Jul 14 15:06:28.903984 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:28.903859 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:28.904354 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:28.904330 2550 patch_prober.go:28] interesting pod/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 15:06:28.904534 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:28.904511 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 15:06:29.903828 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:29.903785 2550 patch_prober.go:28] interesting pod/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 15:06:29.904446 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:29.904410 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 15:06:30.478034 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.478002 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9"] Jul 14 15:06:30.480500 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.480477 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.483425 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.483398 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-03dca1\"/\"openshift-service-ca.crt\"" Jul 14 15:06:30.484603 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.484580 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-03dca1\"/\"test-model-03dca1-kserve-self-signed-certs\"" Jul 14 15:06:30.484805 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.484783 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-03dca1\"/\"kube-root-ca.crt\"" Jul 14 15:06:30.485071 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.485050 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-03dca1\"/\"default-dockercfg-26vm9\"" Jul 14 15:06:30.491556 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.491533 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9"] Jul 14 15:06:30.581740 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.581684 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tls-certs\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.583316 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.581759 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-model-cache\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.583316 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.581897 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tmp-dir\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.583316 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.581978 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxc2d\" (UniqueName: \"kubernetes.io/projected/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-kube-api-access-hxc2d\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.583316 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.582077 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-dshm\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.583316 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.582139 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-home\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683105 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tls-certs\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683386 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683191 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-model-cache\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683386 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683253 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tmp-dir\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683386 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683287 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxc2d\" (UniqueName: \"kubernetes.io/projected/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-kube-api-access-hxc2d\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683386 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683331 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-dshm\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683386 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683369 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-home\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683725 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683698 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-model-cache\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683800 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683745 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tmp-dir\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.683875 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.683853 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-home\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.685469 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.685451 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-dshm\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.685822 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.685802 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tls-certs\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.693141 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.693110 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxc2d\" (UniqueName: \"kubernetes.io/projected/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-kube-api-access-hxc2d\") pod \"test-model-03dca1-kserve-57d84bb45c-n8pg9\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.793198 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.793143 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:30.937403 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.907453 2550 patch_prober.go:28] interesting pod/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 15:06:30.937403 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.907510 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 15:06:30.987941 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:30.987915 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9"] Jul 14 15:06:30.990484 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:06:30.990452 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4c510e5_a967_4ace_89bc_f685b3b9b9bc.slice/crio-7ecbe4571aa63c55419dabe85196b43a0181fee2fe2f8929a03971013534a3ab WatchSource:0}: Error finding container 7ecbe4571aa63c55419dabe85196b43a0181fee2fe2f8929a03971013534a3ab: Status 404 returned error can't find the container with id 7ecbe4571aa63c55419dabe85196b43a0181fee2fe2f8929a03971013534a3ab Jul 14 15:06:31.633424 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:31.633387 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" event={"ID":"d4c510e5-a967-4ace-89bc-f685b3b9b9bc","Type":"ContainerStarted","Data":"7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18"} Jul 14 15:06:31.633610 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:31.633436 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" event={"ID":"d4c510e5-a967-4ace-89bc-f685b3b9b9bc","Type":"ContainerStarted","Data":"7ecbe4571aa63c55419dabe85196b43a0181fee2fe2f8929a03971013534a3ab"} Jul 14 15:06:31.658373 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:31.658322 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" podStartSLOduration=1.6583020419999999 podStartE2EDuration="1.658302042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:06:31.655341671 +0000 UTC m=+2181.090326318" watchObservedRunningTime="2026-07-14 15:06:31.658302042 +0000 UTC m=+2181.093286665" Jul 14 15:06:31.903809 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:31.903695 2550 patch_prober.go:28] interesting pod/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 15:06:31.903809 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:31.903767 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 15:06:32.908081 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:32.908048 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:32.908500 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:32.908348 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:32.908977 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:32.908945 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:06:40.793894 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:40.793849 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:40.793894 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:40.793908 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:40.806907 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:40.806871 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:06:41.696417 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:06:41.696386 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:07:10.525004 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:10.524948 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9"] Jul 14 15:07:25.545248 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.545196 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" podUID="d4c510e5-a967-4ace-89bc-f685b3b9b9bc" containerName="main" containerID="cri-o://7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18" gracePeriod=45 Jul 14 15:07:25.796058 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.795991 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:07:25.900926 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.900890 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tmp-dir\") pod \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " Jul 14 15:07:25.901128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.900935 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-model-cache\") pod \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " Jul 14 15:07:25.901128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901011 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-home\") pod \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " Jul 14 15:07:25.901128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901032 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-dshm\") pod \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " Jul 14 15:07:25.901128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901064 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxc2d\" (UniqueName: \"kubernetes.io/projected/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-kube-api-access-hxc2d\") pod \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " Jul 14 15:07:25.901128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901106 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tls-certs\") pod \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\" (UID: \"d4c510e5-a967-4ace-89bc-f685b3b9b9bc\") " Jul 14 15:07:25.901414 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901243 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4c510e5-a967-4ace-89bc-f685b3b9b9bc" (UID: "d4c510e5-a967-4ace-89bc-f685b3b9b9bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:25.901414 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901254 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4c510e5-a967-4ace-89bc-f685b3b9b9bc" (UID: "d4c510e5-a967-4ace-89bc-f685b3b9b9bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:25.901414 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901264 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-home" (OuterVolumeSpecName: "home") pod "d4c510e5-a967-4ace-89bc-f685b3b9b9bc" (UID: "d4c510e5-a967-4ace-89bc-f685b3b9b9bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:25.901522 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901470 2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tmp-dir\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:25.901522 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901489 2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-model-cache\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:25.901522 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.901501 2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-home\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:25.903314 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.903288 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-kube-api-access-hxc2d" (OuterVolumeSpecName: "kube-api-access-hxc2d") pod "d4c510e5-a967-4ace-89bc-f685b3b9b9bc" (UID: "d4c510e5-a967-4ace-89bc-f685b3b9b9bc"). InnerVolumeSpecName "kube-api-access-hxc2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:07:25.903399 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.903354 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4c510e5-a967-4ace-89bc-f685b3b9b9bc" (UID: "d4c510e5-a967-4ace-89bc-f685b3b9b9bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:07:25.903399 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.903368 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-dshm" (OuterVolumeSpecName: "dshm") pod "d4c510e5-a967-4ace-89bc-f685b3b9b9bc" (UID: "d4c510e5-a967-4ace-89bc-f685b3b9b9bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:25.911234 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.911209 2550 generic.go:356] "Generic (PLEG): container finished" podID="d4c510e5-a967-4ace-89bc-f685b3b9b9bc" containerID="7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18" exitCode=0 Jul 14 15:07:25.911348 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.911264 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" Jul 14 15:07:25.911348 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.911300 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" event={"ID":"d4c510e5-a967-4ace-89bc-f685b3b9b9bc","Type":"ContainerDied","Data":"7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18"} Jul 14 15:07:25.911463 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.911355 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9" event={"ID":"d4c510e5-a967-4ace-89bc-f685b3b9b9bc","Type":"ContainerDied","Data":"7ecbe4571aa63c55419dabe85196b43a0181fee2fe2f8929a03971013534a3ab"} Jul 14 15:07:25.911463 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.911380 2550 scope.go:117] "RemoveContainer" containerID="7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18" Jul 14 15:07:25.920553 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.920536 2550 scope.go:117] "RemoveContainer" containerID="7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18" Jul 14 15:07:25.920801 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:07:25.920783 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18\": container with ID starting with 7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18 not found: ID does not exist" containerID="7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18" Jul 14 15:07:25.920872 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.920814 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18"} err="failed to get container status \"7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18\": rpc error: code = NotFound desc = could not find container \"7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18\": container with ID starting with 7a917008c0ef1d1566fcdcc3d33af102128f446b19dea79a28a5d53c91dc5b18 not found: ID does not exist" Jul 14 15:07:25.933608 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.933584 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9"] Jul 14 15:07:25.937432 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:25.937412 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-03dca1/test-model-03dca1-kserve-57d84bb45c-n8pg9"] Jul 14 15:07:26.002030 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:26.001993 2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-tls-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:26.002030 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:26.002024 2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-dshm\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:26.002220 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:26.002037 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxc2d\" (UniqueName: \"kubernetes.io/projected/d4c510e5-a967-4ace-89bc-f685b3b9b9bc-kube-api-access-hxc2d\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:27.120624 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:27.120587 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4c510e5-a967-4ace-89bc-f685b3b9b9bc" path="/var/lib/kubelet/pods/d4c510e5-a967-4ace-89bc-f685b3b9b9bc/volumes" Jul 14 15:07:37.457855 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:37.457765 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n"] Jul 14 15:07:37.458326 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:37.458065 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" containerID="cri-o://353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f" gracePeriod=30 Jul 14 15:07:41.149163 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:41.149132 2550 patch_prober.go:28] interesting pod/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 15:07:41.149606 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:41.149186 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 15:07:42.710671 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.710641 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:07:42.758278 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758243 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-data\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758278 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758278 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-credential-socket\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758322 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-certs\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758358 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7rtv\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-kube-api-access-r7rtv\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758377 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-token\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758418 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istiod-ca-cert\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758441 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-podinfo\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758470 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-socket\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758869 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758568 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:42.758869 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758584 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-envoy\") pod \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\" (UID: \"4ce34ba4-7a22-416f-aa50-898f62c2d83a\") " Jul 14 15:07:42.758869 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758690 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:42.759068 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.758880 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:42.759145 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.759118 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:07:42.759209 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.759165 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:42.759209 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.759184 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:42.759209 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.759199 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:42.759594 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.759575 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:42.761184 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.761157 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:07:42.761290 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.761194 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:07:42.761290 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.761206 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:07:42.761372 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.761291 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-kube-api-access-r7rtv" (OuterVolumeSpecName: "kube-api-access-r7rtv") pod "4ce34ba4-7a22-416f-aa50-898f62c2d83a" (UID: "4ce34ba4-7a22-416f-aa50-898f62c2d83a"). InnerVolumeSpecName "kube-api-access-r7rtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:07:42.860350 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.860313 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rtv\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-kube-api-access-r7rtv\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:42.860350 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.860347 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:42.860350 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.860357 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:42.860596 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.860367 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:42.860596 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.860376 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:42.860596 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:42.860384 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce34ba4-7a22-416f-aa50-898f62c2d83a-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:07:43.002015 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.001949 2550 generic.go:356] "Generic (PLEG): container finished" podID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerID="353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f" exitCode=0 Jul 14 15:07:43.002015 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.002008 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" event={"ID":"4ce34ba4-7a22-416f-aa50-898f62c2d83a","Type":"ContainerDied","Data":"353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f"} Jul 14 15:07:43.002015 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.002020 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" Jul 14 15:07:43.002262 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.002101 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n" event={"ID":"4ce34ba4-7a22-416f-aa50-898f62c2d83a","Type":"ContainerDied","Data":"fd87f5f506c97108224d1bdebfc10ee81307d3f6ca469ef92a17c13e06c2ced0"} Jul 14 15:07:43.002262 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.002125 2550 scope.go:117] "RemoveContainer" containerID="353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f" Jul 14 15:07:43.011463 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.011439 2550 scope.go:117] "RemoveContainer" containerID="62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6" Jul 14 15:07:43.019790 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.019768 2550 scope.go:117] "RemoveContainer" containerID="353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f" Jul 14 15:07:43.020072 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:07:43.020053 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f\": container with ID starting with 353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f not found: ID does not exist" containerID="353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f" Jul 14 15:07:43.020128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.020084 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f"} err="failed to get container status \"353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f\": rpc error: code = NotFound desc = could not find container \"353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f\": container with ID starting with 353cc327afc1280d8ed7bd0ac08484091d76f61c1e42313234be82211c61703f not found: ID does not exist" Jul 14 15:07:43.020128 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.020102 2550 scope.go:117] "RemoveContainer" containerID="62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6" Jul 14 15:07:43.020322 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:07:43.020308 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6\": container with ID starting with 62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6 not found: ID does not exist" containerID="62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6" Jul 14 15:07:43.020359 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.020326 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6"} err="failed to get container status \"62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6\": rpc error: code = NotFound desc = could not find container \"62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6\": container with ID starting with 62c66a43348ebf0fc0c50cfff429198dd237c5b088af43e5e5a76e3d63b131e6 not found: ID does not exist" Jul 14 15:07:43.025399 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.025374 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n"] Jul 14 15:07:43.031012 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.030990 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-03dca1-openshift-default-fdcbcfd49-jnt8n"] Jul 14 15:07:43.120565 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:07:43.120532 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" path="/var/lib/kubelet/pods/4ce34ba4-7a22-416f-aa50-898f62c2d83a/volumes" Jul 14 15:08:03.683452 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:03.683410 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml"] Jul 14 15:08:04.106613 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.106571 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" containerID="cri-o://b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f" gracePeriod=30 Jul 14 15:08:04.705465 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.705426 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx"] Jul 14 15:08:04.706045 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.706026 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" Jul 14 15:08:04.706099 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.706047 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" Jul 14 15:08:04.706099 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.706091 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c510e5-a967-4ace-89bc-f685b3b9b9bc" containerName="main" Jul 14 15:08:04.706165 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.706099 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c510e5-a967-4ace-89bc-f685b3b9b9bc" containerName="main" Jul 14 15:08:04.706198 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.706185 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" Jul 14 15:08:04.706232 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.706200 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4c510e5-a967-4ace-89bc-f685b3b9b9bc" containerName="main" Jul 14 15:08:04.709495 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.709477 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.713857 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.713834 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e0a4af-openshift-default-dockercfg-4l7ls\"" Jul 14 15:08:04.728610 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.728588 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx"] Jul 14 15:08:04.864119 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864081 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-certs\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.864119 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864120 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-credential-socket\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.864377 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864143 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-token\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.864377 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864251 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istiod-ca-cert\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.864377 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864292 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-podinfo\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.864377 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864342 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-data\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.864517 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864404 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hf49\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-kube-api-access-9hf49\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.864517 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864453 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-envoy\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.864517 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.864505 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-socket\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965161 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965073 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-certs\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965161 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965110 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-credential-socket\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965161 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965125 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-token\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965466 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965274 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istiod-ca-cert\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965466 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965320 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-podinfo\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965466 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965358 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-data\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965466 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965412 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hf49\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-kube-api-access-9hf49\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965673 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965478 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-envoy\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965673 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965524 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-socket\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965673 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965537 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-credential-socket\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965673 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965476 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-certs\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965905 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965734 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-data\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.965905 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.965854 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-socket\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.966056 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.966033 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istiod-ca-cert\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.967782 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.967754 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-envoy\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.967942 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.967919 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-podinfo\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.976036 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.976011 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-token\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:04.976246 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:04.976228 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hf49\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-kube-api-access-9hf49\") pod \"e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:05.020279 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:05.020243 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:05.172810 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:05.172787 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx"] Jul 14 15:08:05.175005 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:08:05.174977 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5594d46f_8aa1_4e63_b44f_82d7cdd8712e.slice/crio-6074e4f91e6bcece8fb9819e5c68f3fd9cbb7510e8e75788867cedee8b90d5ab WatchSource:0}: Error finding container 6074e4f91e6bcece8fb9819e5c68f3fd9cbb7510e8e75788867cedee8b90d5ab: Status 404 returned error can't find the container with id 6074e4f91e6bcece8fb9819e5c68f3fd9cbb7510e8e75788867cedee8b90d5ab Jul 14 15:08:05.177180 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:05.177148 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:05.177240 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:05.177213 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:05.177273 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:05.177243 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:06.119938 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:06.119901 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" event={"ID":"5594d46f-8aa1-4e63-b44f-82d7cdd8712e","Type":"ContainerStarted","Data":"ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf"} Jul 14 15:08:06.120445 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:06.119950 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" event={"ID":"5594d46f-8aa1-4e63-b44f-82d7cdd8712e","Type":"ContainerStarted","Data":"6074e4f91e6bcece8fb9819e5c68f3fd9cbb7510e8e75788867cedee8b90d5ab"} Jul 14 15:08:06.143549 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:06.143497 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podStartSLOduration=2.143480181 podStartE2EDuration="2.143480181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:08:06.141843427 +0000 UTC m=+2275.576828052" watchObservedRunningTime="2026-07-14 15:08:06.143480181 +0000 UTC m=+2275.578464805" Jul 14 15:08:07.021100 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:07.021063 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:07.025949 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:07.025923 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:07.125316 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:07.125285 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:07.126159 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:07.126138 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:09.371335 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.371271 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:08:09.507578 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507541 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istiod-ca-cert\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.507778 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507588 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-envoy\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.507778 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507627 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-certs\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.507778 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507682 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-data\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.507778 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507711 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9ncj\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-kube-api-access-m9ncj\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.507778 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507739 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-socket\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.508085 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507782 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-token\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.508085 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507824 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-credential-socket\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.508085 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507872 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-podinfo\") pod \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\" (UID: \"7154aea7-a868-43ab-acd5-ef45ad0ea6f0\") " Jul 14 15:08:09.508085 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.507982 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:08:09.508306 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.508094 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:08:09.508306 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.508105 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:08:09.508306 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.508229 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:09.508306 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.508248 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:09.508306 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.508261 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:09.508573 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.508329 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:08:09.508706 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.508676 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:08:09.510052 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.510024 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:08:09.510174 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.510115 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:08:09.510218 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.510164 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-kube-api-access-m9ncj" (OuterVolumeSpecName: "kube-api-access-m9ncj") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "kube-api-access-m9ncj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:08:09.510218 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.510195 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7154aea7-a868-43ab-acd5-ef45ad0ea6f0" (UID: "7154aea7-a868-43ab-acd5-ef45ad0ea6f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:08:09.609615 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.609571 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:09.609615 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.609606 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:09.609615 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.609619 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9ncj\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-kube-api-access-m9ncj\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:09.609847 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.609633 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:09.609847 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.609645 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:09.609847 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:09.609656 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7154aea7-a868-43ab-acd5-ef45ad0ea6f0-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:08:10.142679 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.142647 2550 generic.go:356] "Generic (PLEG): container finished" podID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerID="b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f" exitCode=0 Jul 14 15:08:10.142856 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.142699 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" Jul 14 15:08:10.142856 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.142725 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" event={"ID":"7154aea7-a868-43ab-acd5-ef45ad0ea6f0","Type":"ContainerDied","Data":"b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f"} Jul 14 15:08:10.142856 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.142775 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml" event={"ID":"7154aea7-a868-43ab-acd5-ef45ad0ea6f0","Type":"ContainerDied","Data":"9171add066cb89eec475f4aa57b1202a2d9118ceaa44ac12afd77ef55f9fad12"} Jul 14 15:08:10.142856 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.142798 2550 scope.go:117] "RemoveContainer" containerID="b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f" Jul 14 15:08:10.152653 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.152633 2550 scope.go:117] "RemoveContainer" containerID="c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74" Jul 14 15:08:10.160687 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.160672 2550 scope.go:117] "RemoveContainer" containerID="b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f" Jul 14 15:08:10.160944 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:08:10.160927 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f\": container with ID starting with b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f not found: ID does not exist" containerID="b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f" Jul 14 15:08:10.161022 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.160951 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f"} err="failed to get container status \"b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f\": rpc error: code = NotFound desc = could not find container \"b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f\": container with ID starting with b196d37a0d3ecfba6f2007ffb050cae9e9a69bc030ec5208e35f503d5048481f not found: ID does not exist" Jul 14 15:08:10.161022 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.160988 2550 scope.go:117] "RemoveContainer" containerID="c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74" Jul 14 15:08:10.161221 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:08:10.161203 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74\": container with ID starting with c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74 not found: ID does not exist" containerID="c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74" Jul 14 15:08:10.161271 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.161229 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74"} err="failed to get container status \"c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74\": rpc error: code = NotFound desc = could not find container \"c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74\": container with ID starting with c692e840e4024177e1f6b107bc84dc713952eb7d1e1732cd2757cf4bfe765b74 not found: ID does not exist" Jul 14 15:08:10.167219 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.167191 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml"] Jul 14 15:08:10.172202 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:10.172180 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4226a2-openshift-default-586648cc6f-v49ml"] Jul 14 15:08:11.120597 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:11.120568 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" path="/var/lib/kubelet/pods/7154aea7-a868-43ab-acd5-ef45ad0ea6f0/volumes" Jul 14 15:08:14.243383 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.243349 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg"] Jul 14 15:08:14.243977 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.243734 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" Jul 14 15:08:14.243977 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.243744 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" Jul 14 15:08:14.243977 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.243752 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" Jul 14 15:08:14.243977 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.243758 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" Jul 14 15:08:14.243977 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.243822 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" Jul 14 15:08:14.243977 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.243833 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce34ba4-7a22-416f-aa50-898f62c2d83a" containerName="istio-proxy" Jul 14 15:08:14.248651 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.248630 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.254057 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.254035 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-40b1bd-openshift-default-dockercfg-hxpt4\"" Jul 14 15:08:14.262113 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.262088 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg"] Jul 14 15:08:14.350689 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.350652 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-podinfo\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.350896 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.350701 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-credential-socket\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.350896 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.350730 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-socket\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.350896 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.350847 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-token\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.351094 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.350898 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmmn2\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-kube-api-access-kmmn2\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.351094 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.351031 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/397a9055-78f1-4bff-abde-b8ce11ffedcb-istiod-ca-cert\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.351094 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.351068 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-certs\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.351252 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.351128 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-envoy\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.351252 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.351206 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-data\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.451845 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.451807 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-data\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.451845 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.451852 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-podinfo\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452116 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.451873 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-credential-socket\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452116 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452004 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-socket\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452116 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452062 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-token\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452116 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452110 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmmn2\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-kube-api-access-kmmn2\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452334 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452226 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/397a9055-78f1-4bff-abde-b8ce11ffedcb-istiod-ca-cert\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452334 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452255 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-credential-socket\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452334 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452310 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-data\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452493 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452336 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-socket\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452600 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452579 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-certs\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452600 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452582 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-certs\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452750 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452642 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-envoy\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.452871 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.452851 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/397a9055-78f1-4bff-abde-b8ce11ffedcb-istiod-ca-cert\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.454523 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.454498 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-podinfo\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.454632 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.454565 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-envoy\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.460955 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.460930 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-token\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.461713 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.461691 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmmn2\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-kube-api-access-kmmn2\") pod \"e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.560300 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.560266 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:14.762265 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.762238 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg"] Jul 14 15:08:14.763774 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:08:14.763739 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397a9055_78f1_4bff_abde_b8ce11ffedcb.slice/crio-c511c0cf7c5faebd39d153aedfaf341c40c17afab36d65f411f763046ea0056d WatchSource:0}: Error finding container c511c0cf7c5faebd39d153aedfaf341c40c17afab36d65f411f763046ea0056d: Status 404 returned error can't find the container with id c511c0cf7c5faebd39d153aedfaf341c40c17afab36d65f411f763046ea0056d Jul 14 15:08:14.766080 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.766048 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:14.766154 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.766116 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:14.766154 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:14.766144 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:15.170381 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:15.170282 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" event={"ID":"397a9055-78f1-4bff-abde-b8ce11ffedcb","Type":"ContainerStarted","Data":"b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f"} Jul 14 15:08:15.170381 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:15.170341 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" event={"ID":"397a9055-78f1-4bff-abde-b8ce11ffedcb","Type":"ContainerStarted","Data":"c511c0cf7c5faebd39d153aedfaf341c40c17afab36d65f411f763046ea0056d"} Jul 14 15:08:15.192777 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:15.192725 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" podStartSLOduration=1.192707504 podStartE2EDuration="1.192707504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:08:15.19088234 +0000 UTC m=+2284.625866965" watchObservedRunningTime="2026-07-14 15:08:15.192707504 +0000 UTC m=+2284.627692127" Jul 14 15:08:15.561455 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:15.561416 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:15.566520 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:15.566494 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:16.177627 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:16.177592 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:16.178414 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:16.178397 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:21.206457 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:21.206375 2550 generic.go:356] "Generic (PLEG): container finished" podID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerID="ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf" exitCode=0 Jul 14 15:08:21.206457 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:21.206414 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" event={"ID":"5594d46f-8aa1-4e63-b44f-82d7cdd8712e","Type":"ContainerDied","Data":"ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf"} Jul 14 15:08:21.206938 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:21.206788 2550 scope.go:117] "RemoveContainer" containerID="ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf" Jul 14 15:08:21.207811 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:21.207777 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:21.207904 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:21.207852 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:21.207904 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:21.207888 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:22.125660 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:22.125615 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:22.213141 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:22.213103 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" event={"ID":"5594d46f-8aa1-4e63-b44f-82d7cdd8712e","Type":"ContainerStarted","Data":"8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a"} Jul 14 15:08:23.020905 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:23.020865 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:23.022145 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:23.022117 2550 patch_prober.go:28] interesting pod/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 15:08:23.022300 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:23.022172 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 15:08:24.021549 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:24.021510 2550 patch_prober.go:28] interesting pod/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 15:08:24.022088 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:24.021571 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 15:08:25.021118 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:25.021066 2550 patch_prober.go:28] interesting pod/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 15:08:25.021329 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:25.021147 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 15:08:26.020829 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:26.020790 2550 patch_prober.go:28] interesting pod/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 15:08:26.021296 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:26.020847 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 15:08:27.020872 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:27.020833 2550 patch_prober.go:28] interesting pod/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 15:08:27.021501 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:27.021466 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 15:08:28.021175 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:28.021130 2550 patch_prober.go:28] interesting pod/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 15:08:28.021801 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:28.021766 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 15:08:29.025490 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:29.025453 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:29.025921 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:29.025756 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:29.026395 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:29.026375 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:08:32.219131 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:32.219083 2550 patch_prober.go:28] interesting pod/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 15:08:32.219564 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:32.219160 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 15:08:34.278728 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:34.278688 2550 generic.go:356] "Generic (PLEG): container finished" podID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerID="b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f" exitCode=0 Jul 14 15:08:34.279220 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:34.278763 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" event={"ID":"397a9055-78f1-4bff-abde-b8ce11ffedcb","Type":"ContainerDied","Data":"b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f"} Jul 14 15:08:34.279220 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:34.279157 2550 scope.go:117] "RemoveContainer" containerID="b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f" Jul 14 15:08:34.280105 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:34.280077 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:34.280200 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:34.280140 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:34.280200 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:34.280172 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:08:35.285788 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:35.285751 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" event={"ID":"397a9055-78f1-4bff-abde-b8ce11ffedcb","Type":"ContainerStarted","Data":"33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc"} Jul 14 15:08:35.560472 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:35.560374 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:35.560726 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:35.560701 2550 patch_prober.go:28] interesting pod/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 14 15:08:35.560779 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:35.560757 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 14 15:08:36.561439 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:36.561397 2550 patch_prober.go:28] interesting pod/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 14 15:08:36.561835 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:36.561461 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 14 15:08:37.561343 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:37.561305 2550 patch_prober.go:28] interesting pod/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 14 15:08:37.561526 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:37.561369 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 14 15:08:38.560924 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:38.560893 2550 patch_prober.go:28] interesting pod/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 14 15:08:38.561126 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:38.560945 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 14 15:08:39.564940 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:39.564910 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:39.565406 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:39.565246 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:39.565835 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:39.565817 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:08:56.420217 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:56.420182 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx"] Jul 14 15:08:56.420779 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:56.420400 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" containerID="cri-o://8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a" gracePeriod=30 Jul 14 15:08:59.027896 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:59.027855 2550 patch_prober.go:28] interesting pod/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 15:08:59.028308 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:08:59.027937 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 15:09:01.673005 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.672949 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:09:01.807694 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.807658 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-certs\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.807897 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.807719 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istiod-ca-cert\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.807897 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.807757 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-data\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.807897 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.807777 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-socket\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.807897 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.807809 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-credential-socket\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.807897 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.807834 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-envoy\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.807897 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.807893 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hf49\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-kube-api-access-9hf49\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.808269 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.807982 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-token\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.808269 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808021 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-podinfo\") pod \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\" (UID: \"5594d46f-8aa1-4e63-b44f-82d7cdd8712e\") " Jul 14 15:09:01.808269 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808052 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:01.808269 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808184 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:01.808269 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808204 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:01.808538 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808321 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:09:01.808538 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808374 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:01.808538 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808392 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:01.808538 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808408 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:01.808812 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.808790 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-data" (OuterVolumeSpecName: "istio-data") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:01.810368 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.810345 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:09:01.810470 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.810391 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-kube-api-access-9hf49" (OuterVolumeSpecName: "kube-api-access-9hf49") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "kube-api-access-9hf49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:09:01.810657 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.810632 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-token" (OuterVolumeSpecName: "istio-token") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:09:01.810759 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.810627 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5594d46f-8aa1-4e63-b44f-82d7cdd8712e" (UID: "5594d46f-8aa1-4e63-b44f-82d7cdd8712e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:01.909225 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.909190 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:01.909225 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.909220 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:01.909225 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.909229 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:01.909467 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.909237 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:01.909467 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.909245 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:01.909467 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:01.909253 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hf49\" (UniqueName: \"kubernetes.io/projected/5594d46f-8aa1-4e63-b44f-82d7cdd8712e-kube-api-access-9hf49\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:02.442211 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.442176 2550 generic.go:356] "Generic (PLEG): container finished" podID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerID="8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a" exitCode=0 Jul 14 15:09:02.442372 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.442223 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" Jul 14 15:09:02.442372 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.442254 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" event={"ID":"5594d46f-8aa1-4e63-b44f-82d7cdd8712e","Type":"ContainerDied","Data":"8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a"} Jul 14 15:09:02.442372 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.442308 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx" event={"ID":"5594d46f-8aa1-4e63-b44f-82d7cdd8712e","Type":"ContainerDied","Data":"6074e4f91e6bcece8fb9819e5c68f3fd9cbb7510e8e75788867cedee8b90d5ab"} Jul 14 15:09:02.442372 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.442330 2550 scope.go:117] "RemoveContainer" containerID="8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a" Jul 14 15:09:02.451976 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.451945 2550 scope.go:117] "RemoveContainer" containerID="ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf" Jul 14 15:09:02.460069 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.460048 2550 scope.go:117] "RemoveContainer" containerID="8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a" Jul 14 15:09:02.460345 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:09:02.460327 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a\": container with ID starting with 8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a not found: ID does not exist" containerID="8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a" Jul 14 15:09:02.460394 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.460355 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a"} err="failed to get container status \"8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a\": rpc error: code = NotFound desc = could not find container \"8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a\": container with ID starting with 8b46689d8cf300f53d4d3b7d8c00c3edee71ecb4fd22b8784795a4a8a612157a not found: ID does not exist" Jul 14 15:09:02.460394 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.460373 2550 scope.go:117] "RemoveContainer" containerID="ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf" Jul 14 15:09:02.460658 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:09:02.460642 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf\": container with ID starting with ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf not found: ID does not exist" containerID="ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf" Jul 14 15:09:02.460729 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.460663 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf"} err="failed to get container status \"ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf\": rpc error: code = NotFound desc = could not find container \"ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf\": container with ID starting with ceb16b0f28353901eace076df6a7a69a7e0b1dee4ea1dcaeaf7b1524e7031ddf not found: ID does not exist" Jul 14 15:09:02.467356 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.467329 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx"] Jul 14 15:09:02.471233 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:02.471209 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e0a4af-openshift-default-5c78cb579c-xjgdx"] Jul 14 15:09:03.120676 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:03.120576 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" path="/var/lib/kubelet/pods/5594d46f-8aa1-4e63-b44f-82d7cdd8712e/volumes" Jul 14 15:09:22.613051 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:22.613013 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg"] Jul 14 15:09:22.613433 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:22.613283 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" containerID="cri-o://33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc" gracePeriod=30 Jul 14 15:09:23.523774 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.523741 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc"] Jul 14 15:09:23.524267 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.524238 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" Jul 14 15:09:23.524267 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.524269 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" Jul 14 15:09:23.524371 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.524282 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" Jul 14 15:09:23.524371 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.524288 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" Jul 14 15:09:23.524371 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.524294 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" Jul 14 15:09:23.524371 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.524299 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" Jul 14 15:09:23.524371 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.524360 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" Jul 14 15:09:23.524371 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.524367 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7154aea7-a868-43ab-acd5-ef45ad0ea6f0" containerName="istio-proxy" Jul 14 15:09:23.527309 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.527289 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.532635 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.532608 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ebb1a4-openshift-default-dockercfg-z25kk\"" Jul 14 15:09:23.543555 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.543532 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc"] Jul 14 15:09:23.583418 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583389 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2qm\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-kube-api-access-vx2qm\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.583577 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583441 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-certs\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.583577 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583469 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a624859-5c7e-4b21-a263-7ca7ac650f40-istiod-ca-cert\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.583577 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583515 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-envoy\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.583577 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583530 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-podinfo\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.583577 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583556 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-socket\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.583577 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583572 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-credential-socket\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.583790 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583664 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-data\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.583790 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.583696 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-token\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.684988 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.684920 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-certs\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.684988 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.684998 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a624859-5c7e-4b21-a263-7ca7ac650f40-istiod-ca-cert\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685057 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-envoy\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685165 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-podinfo\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685200 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-socket\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685220 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-credential-socket\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685269 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-data\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685311 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-token\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685359 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-certs\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685363 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2qm\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-kube-api-access-vx2qm\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685533 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-socket\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.685723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685668 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-data\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.686120 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685739 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-credential-socket\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.686120 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.685764 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a624859-5c7e-4b21-a263-7ca7ac650f40-istiod-ca-cert\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.687502 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.687479 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-envoy\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.687595 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.687578 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-podinfo\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.694296 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.694273 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-token\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.695074 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.695049 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2qm\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-kube-api-access-vx2qm\") pod \"e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:23.837920 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:23.837808 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:24.009226 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.009182 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc"] Jul 14 15:09:24.010384 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:09:24.010355 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a624859_5c7e_4b21_a263_7ca7ac650f40.slice/crio-f6a99e26a38d6d1370ac265a7c309a971ec1dc87834529f58d7a26247af8e842 WatchSource:0}: Error finding container f6a99e26a38d6d1370ac265a7c309a971ec1dc87834529f58d7a26247af8e842: Status 404 returned error can't find the container with id f6a99e26a38d6d1370ac265a7c309a971ec1dc87834529f58d7a26247af8e842 Jul 14 15:09:24.012715 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.012678 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:24.012827 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.012760 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:24.012827 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.012802 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:24.557203 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.557170 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" event={"ID":"3a624859-5c7e-4b21-a263-7ca7ac650f40","Type":"ContainerStarted","Data":"f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177"} Jul 14 15:09:24.557382 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.557219 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" event={"ID":"3a624859-5c7e-4b21-a263-7ca7ac650f40","Type":"ContainerStarted","Data":"f6a99e26a38d6d1370ac265a7c309a971ec1dc87834529f58d7a26247af8e842"} Jul 14 15:09:24.567804 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.567778 2550 patch_prober.go:28] interesting pod/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 15:09:24.567927 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.567825 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 15:09:24.596884 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.596823 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" podStartSLOduration=1.5968052190000002 podStartE2EDuration="1.596805219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:09:24.594655733 +0000 UTC m=+2354.029640368" watchObservedRunningTime="2026-07-14 15:09:24.596805219 +0000 UTC m=+2354.031789845" Jul 14 15:09:24.839057 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:24.838956 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:25.843848 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:25.843817 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:26.567507 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:26.567477 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:26.568512 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:26.568488 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:27.876437 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.876374 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:09:27.926695 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926665 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmmn2\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-kube-api-access-kmmn2\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.926879 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926713 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-socket\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.926879 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926742 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-data\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.926879 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926773 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-envoy\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.926879 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926811 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-certs\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.926879 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926845 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-token\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.926879 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926879 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/397a9055-78f1-4bff-abde-b8ce11ffedcb-istiod-ca-cert\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.927210 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926908 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-podinfo\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.927210 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.926926 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-credential-socket\") pod \"397a9055-78f1-4bff-abde-b8ce11ffedcb\" (UID: \"397a9055-78f1-4bff-abde-b8ce11ffedcb\") " Jul 14 15:09:27.927210 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.927026 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:27.927210 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.927202 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:27.927426 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.927386 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:27.927505 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.927479 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397a9055-78f1-4bff-abde-b8ce11ffedcb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:09:27.927566 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.927539 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:27.927777 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.927734 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-data" (OuterVolumeSpecName: "istio-data") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:27.929191 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.929158 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-kube-api-access-kmmn2" (OuterVolumeSpecName: "kube-api-access-kmmn2") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "kube-api-access-kmmn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:09:27.929422 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.929404 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:09:27.929625 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.929604 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:09:27.929755 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:27.929731 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-token" (OuterVolumeSpecName: "istio-token") pod "397a9055-78f1-4bff-abde-b8ce11ffedcb" (UID: "397a9055-78f1-4bff-abde-b8ce11ffedcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:09:28.028675 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.028633 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:28.028675 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.028669 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:28.028675 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.028678 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/397a9055-78f1-4bff-abde-b8ce11ffedcb-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:28.028675 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.028685 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:28.029029 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.028694 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:28.029029 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.028704 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmmn2\" (UniqueName: \"kubernetes.io/projected/397a9055-78f1-4bff-abde-b8ce11ffedcb-kube-api-access-kmmn2\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:28.029029 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.028712 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:28.029029 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.028720 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/397a9055-78f1-4bff-abde-b8ce11ffedcb-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:09:28.578651 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.578614 2550 generic.go:356] "Generic (PLEG): container finished" podID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerID="33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc" exitCode=0 Jul 14 15:09:28.578838 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.578677 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" Jul 14 15:09:28.578838 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.578702 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" event={"ID":"397a9055-78f1-4bff-abde-b8ce11ffedcb","Type":"ContainerDied","Data":"33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc"} Jul 14 15:09:28.578838 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.578786 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg" event={"ID":"397a9055-78f1-4bff-abde-b8ce11ffedcb","Type":"ContainerDied","Data":"c511c0cf7c5faebd39d153aedfaf341c40c17afab36d65f411f763046ea0056d"} Jul 14 15:09:28.578943 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.578809 2550 scope.go:117] "RemoveContainer" containerID="33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc" Jul 14 15:09:28.588326 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.588307 2550 scope.go:117] "RemoveContainer" containerID="b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f" Jul 14 15:09:28.598990 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.598947 2550 scope.go:117] "RemoveContainer" containerID="33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc" Jul 14 15:09:28.599508 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:09:28.599383 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc\": container with ID starting with 33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc not found: ID does not exist" containerID="33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc" Jul 14 15:09:28.599508 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.599422 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc"} err="failed to get container status \"33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc\": rpc error: code = NotFound desc = could not find container \"33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc\": container with ID starting with 33a9564cdebab851a4b7183574e59f8d7bd751d9fa5912832a8fac137eab53bc not found: ID does not exist" Jul 14 15:09:28.599508 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.599445 2550 scope.go:117] "RemoveContainer" containerID="b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f" Jul 14 15:09:28.599784 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:09:28.599762 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f\": container with ID starting with b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f not found: ID does not exist" containerID="b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f" Jul 14 15:09:28.599828 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.599792 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f"} err="failed to get container status \"b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f\": rpc error: code = NotFound desc = could not find container \"b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f\": container with ID starting with b86cb64fca4b3e2fd73fe804833b26dcbe85376d9cb3bcd74723c5010a78c78f not found: ID does not exist" Jul 14 15:09:28.601587 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.601567 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg"] Jul 14 15:09:28.605679 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:28.605655 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40b1bd-openshift-default-6b9d766d4c-kwjtg"] Jul 14 15:09:29.120788 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:29.120750 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" path="/var/lib/kubelet/pods/397a9055-78f1-4bff-abde-b8ce11ffedcb/volumes" Jul 14 15:09:31.719683 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.719648 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974"] Jul 14 15:09:31.720074 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.720058 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" Jul 14 15:09:31.720074 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.720071 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" Jul 14 15:09:31.720167 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.720157 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" Jul 14 15:09:31.720200 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.720173 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5594d46f-8aa1-4e63-b44f-82d7cdd8712e" containerName="istio-proxy" Jul 14 15:09:31.724619 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.724600 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.727558 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.727538 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-747ac7-openshift-default-dockercfg-jtgwn\"" Jul 14 15:09:31.740763 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.740735 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974"] Jul 14 15:09:31.762720 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.762688 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-credential-socket\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.762864 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.762813 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-envoy\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.762864 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.762855 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c5b997-f156-4b67-8495-295893791b9e-istio-podinfo\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.763015 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.762999 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-data\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.763058 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.763033 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-socket\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.763058 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.763050 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-certs\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.763153 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.763071 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c5b997-f156-4b67-8495-295893791b9e-istiod-ca-cert\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.763153 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.763113 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-istio-token\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.763258 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.763161 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjg5\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-kube-api-access-ncjg5\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.863844 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.863804 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-envoy\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.863844 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.863848 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c5b997-f156-4b67-8495-295893791b9e-istio-podinfo\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864120 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.863918 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-data\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864120 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.863952 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-socket\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864120 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864003 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-certs\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864285 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864165 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c5b997-f156-4b67-8495-295893791b9e-istiod-ca-cert\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864364 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864330 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-certs\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864520 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864403 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-socket\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864601 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864572 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-data\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864689 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864657 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-istio-token\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864755 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864707 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjg5\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-kube-api-access-ncjg5\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864814 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864760 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-credential-socket\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.864887 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.864849 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c5b997-f156-4b67-8495-295893791b9e-istiod-ca-cert\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.865305 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.865282 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-credential-socket\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.866585 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.866563 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-envoy\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.866771 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.866753 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c5b997-f156-4b67-8495-295893791b9e-istio-podinfo\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.875095 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.875008 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjg5\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-kube-api-access-ncjg5\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:31.875852 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:31.875826 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-istio-token\") pod \"e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:32.035450 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:32.035409 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:32.221308 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:32.221063 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974"] Jul 14 15:09:32.226986 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:32.226929 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:32.227079 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:32.227040 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:32.227079 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:32.227075 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:32.606752 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:32.606706 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" event={"ID":"e1c5b997-f156-4b67-8495-295893791b9e","Type":"ContainerStarted","Data":"7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36"} Jul 14 15:09:32.607102 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:32.606779 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" event={"ID":"e1c5b997-f156-4b67-8495-295893791b9e","Type":"ContainerStarted","Data":"cc9576c8353ee591a348f594c734734816bc094057fd286047380cd6bed3cd91"} Jul 14 15:09:32.642716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:32.642662 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" podStartSLOduration=1.642645539 podStartE2EDuration="1.642645539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:09:32.64072735 +0000 UTC m=+2362.075711975" watchObservedRunningTime="2026-07-14 15:09:32.642645539 +0000 UTC m=+2362.077630162" Jul 14 15:09:33.036429 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:33.036387 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:33.041763 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:33.041736 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:33.616145 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:33.616107 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:33.620438 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:33.620404 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:39.657888 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:39.657851 2550 generic.go:356] "Generic (PLEG): container finished" podID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerID="f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177" exitCode=0 Jul 14 15:09:39.658443 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:39.657927 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" event={"ID":"3a624859-5c7e-4b21-a263-7ca7ac650f40","Type":"ContainerDied","Data":"f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177"} Jul 14 15:09:39.658515 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:39.658462 2550 scope.go:117] "RemoveContainer" containerID="f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177" Jul 14 15:09:39.659489 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:39.659460 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:39.659594 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:39.659527 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:39.659594 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:39.659560 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:40.665103 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:40.665070 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" event={"ID":"3a624859-5c7e-4b21-a263-7ca7ac650f40","Type":"ContainerStarted","Data":"d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8"} Jul 14 15:09:40.838871 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:40.838827 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:40.840169 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:40.840123 2550 patch_prober.go:28] interesting pod/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 14 15:09:40.840330 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:40.840202 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 14 15:09:41.568272 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:41.568236 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:41.838924 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:41.838811 2550 patch_prober.go:28] interesting pod/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 14 15:09:41.838924 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:41.838898 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 14 15:09:42.839720 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:42.839274 2550 patch_prober.go:28] interesting pod/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 14 15:09:42.839720 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:42.839343 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 14 15:09:43.838768 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:43.838720 2550 patch_prober.go:28] interesting pod/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 14 15:09:43.839101 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:43.839063 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 14 15:09:44.842708 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:44.842674 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:44.843730 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:44.843713 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:09:47.701851 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:47.701819 2550 generic.go:356] "Generic (PLEG): container finished" podID="e1c5b997-f156-4b67-8495-295893791b9e" containerID="7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36" exitCode=0 Jul 14 15:09:47.702299 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:47.701882 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" event={"ID":"e1c5b997-f156-4b67-8495-295893791b9e","Type":"ContainerDied","Data":"7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36"} Jul 14 15:09:47.702378 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:47.702361 2550 scope.go:117] "RemoveContainer" containerID="7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36" Jul 14 15:09:47.703445 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:47.703412 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:47.703521 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:47.703504 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:47.703569 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:47.703545 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 15:09:48.616924 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:48.616877 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:48.717545 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:48.713947 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" event={"ID":"e1c5b997-f156-4b67-8495-295893791b9e","Type":"ContainerStarted","Data":"7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c"} Jul 14 15:09:49.036178 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:49.036138 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:49.037422 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:49.037391 2550 patch_prober.go:28] interesting pod/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 14 15:09:49.037550 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:49.037445 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 14 15:09:50.036214 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:50.036167 2550 patch_prober.go:28] interesting pod/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 14 15:09:50.036716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:50.036233 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 14 15:09:51.036442 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:51.036405 2550 patch_prober.go:28] interesting pod/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 14 15:09:51.036858 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:51.036465 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 14 15:09:52.036682 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:52.036641 2550 patch_prober.go:28] interesting pod/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 14 15:09:52.037127 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:52.036709 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 14 15:09:53.040334 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:53.040301 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:53.040746 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:53.040724 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:53.041447 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:53.041430 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:09:59.461594 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.461550 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb"] Jul 14 15:09:59.462199 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.462161 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" Jul 14 15:09:59.462199 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.462183 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" Jul 14 15:09:59.462344 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.462309 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="397a9055-78f1-4bff-abde-b8ce11ffedcb" containerName="istio-proxy" Jul 14 15:09:59.464519 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.464498 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.467413 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.467390 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ebb1a4\"/\"kube-root-ca.crt\"" Jul 14 15:09:59.468728 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.468705 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ebb1a4\"/\"openshift-service-ca.crt\"" Jul 14 15:09:59.468861 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.468758 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ebb1a4\"/\"default-dockercfg-dldgp\"" Jul 14 15:09:59.468926 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.468855 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ebb1a4\"/\"auth-test-model-ebb1a4-kserve-self-signed-certs\"" Jul 14 15:09:59.476532 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.476512 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb"] Jul 14 15:09:59.644248 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.644210 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-tmp-dir\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.644448 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.644264 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7flbk\" (UniqueName: \"kubernetes.io/projected/7e673909-83a4-48bf-8e92-1ac1dc71bede-kube-api-access-7flbk\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.644448 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.644364 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e673909-83a4-48bf-8e92-1ac1dc71bede-tls-certs\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.644448 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.644412 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-dshm\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.644602 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.644451 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-model-cache\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.644602 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.644497 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-home\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745253 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745152 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7flbk\" (UniqueName: \"kubernetes.io/projected/7e673909-83a4-48bf-8e92-1ac1dc71bede-kube-api-access-7flbk\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745253 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745218 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e673909-83a4-48bf-8e92-1ac1dc71bede-tls-certs\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745253 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745252 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-dshm\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745530 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745273 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-model-cache\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745530 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745311 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-home\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745530 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745400 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-tmp-dir\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745777 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745750 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-model-cache\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745893 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745786 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-home\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.745893 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.745821 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-tmp-dir\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.747576 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.747555 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-dshm\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.747744 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.747725 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e673909-83a4-48bf-8e92-1ac1dc71bede-tls-certs\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.755474 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.755452 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7flbk\" (UniqueName: \"kubernetes.io/projected/7e673909-83a4-48bf-8e92-1ac1dc71bede-kube-api-access-7flbk\") pod \"auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.774691 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.774663 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:09:59.920025 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:09:59.919998 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb"] Jul 14 15:09:59.922245 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:09:59.922214 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e673909_83a4_48bf_8e92_1ac1dc71bede.slice/crio-c4ce701036a7c610c51ad2098738f10ed8a30db9f44661a4f719a4de1fa0f852 WatchSource:0}: Error finding container c4ce701036a7c610c51ad2098738f10ed8a30db9f44661a4f719a4de1fa0f852: Status 404 returned error can't find the container with id c4ce701036a7c610c51ad2098738f10ed8a30db9f44661a4f719a4de1fa0f852 Jul 14 15:10:00.778046 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:00.778009 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" event={"ID":"7e673909-83a4-48bf-8e92-1ac1dc71bede","Type":"ContainerStarted","Data":"19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c"} Jul 14 15:10:00.778445 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:00.778056 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" event={"ID":"7e673909-83a4-48bf-8e92-1ac1dc71bede","Type":"ContainerStarted","Data":"c4ce701036a7c610c51ad2098738f10ed8a30db9f44661a4f719a4de1fa0f852"} Jul 14 15:10:00.798343 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:00.798288 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" podStartSLOduration=1.79827141 podStartE2EDuration="1.79827141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:10:00.796123993 +0000 UTC m=+2390.231108620" watchObservedRunningTime="2026-07-14 15:10:00.79827141 +0000 UTC m=+2390.233256083" Jul 14 15:10:09.775634 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:09.775599 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:10:09.776058 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:09.775866 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:10:09.787647 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:09.787621 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:10:09.836139 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:09.836111 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:10:11.790689 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:11.790656 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 15:10:11.798809 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:10:11.798788 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 15:12:16.950487 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:16.950394 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb"] Jul 14 15:12:31.971295 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:31.971249 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" podUID="7e673909-83a4-48bf-8e92-1ac1dc71bede" containerName="main" containerID="cri-o://19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c" gracePeriod=45 Jul 14 15:12:32.237174 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.237148 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:12:32.308638 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.308600 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-home\") pod \"7e673909-83a4-48bf-8e92-1ac1dc71bede\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " Jul 14 15:12:32.308841 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.308649 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-tmp-dir\") pod \"7e673909-83a4-48bf-8e92-1ac1dc71bede\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " Jul 14 15:12:32.308841 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.308678 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7flbk\" (UniqueName: \"kubernetes.io/projected/7e673909-83a4-48bf-8e92-1ac1dc71bede-kube-api-access-7flbk\") pod \"7e673909-83a4-48bf-8e92-1ac1dc71bede\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " Jul 14 15:12:32.308841 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.308742 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e673909-83a4-48bf-8e92-1ac1dc71bede-tls-certs\") pod \"7e673909-83a4-48bf-8e92-1ac1dc71bede\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " Jul 14 15:12:32.309067 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.308897 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-model-cache\") pod \"7e673909-83a4-48bf-8e92-1ac1dc71bede\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " Jul 14 15:12:32.309067 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.308941 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-dshm\") pod \"7e673909-83a4-48bf-8e92-1ac1dc71bede\" (UID: \"7e673909-83a4-48bf-8e92-1ac1dc71bede\") " Jul 14 15:12:32.309067 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.308976 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e673909-83a4-48bf-8e92-1ac1dc71bede" (UID: "7e673909-83a4-48bf-8e92-1ac1dc71bede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:12:32.309067 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.309052 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-home" (OuterVolumeSpecName: "home") pod "7e673909-83a4-48bf-8e92-1ac1dc71bede" (UID: "7e673909-83a4-48bf-8e92-1ac1dc71bede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:12:32.309250 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.309210 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e673909-83a4-48bf-8e92-1ac1dc71bede" (UID: "7e673909-83a4-48bf-8e92-1ac1dc71bede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:12:32.309335 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.309321 2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-model-cache\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:12:32.309372 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.309338 2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-home\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:12:32.309372 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.309348 2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-tmp-dir\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:12:32.311161 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.311131 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-dshm" (OuterVolumeSpecName: "dshm") pod "7e673909-83a4-48bf-8e92-1ac1dc71bede" (UID: "7e673909-83a4-48bf-8e92-1ac1dc71bede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:12:32.311376 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.311355 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e673909-83a4-48bf-8e92-1ac1dc71bede-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e673909-83a4-48bf-8e92-1ac1dc71bede" (UID: "7e673909-83a4-48bf-8e92-1ac1dc71bede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:12:32.311429 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.311371 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e673909-83a4-48bf-8e92-1ac1dc71bede-kube-api-access-7flbk" (OuterVolumeSpecName: "kube-api-access-7flbk") pod "7e673909-83a4-48bf-8e92-1ac1dc71bede" (UID: "7e673909-83a4-48bf-8e92-1ac1dc71bede"). InnerVolumeSpecName "kube-api-access-7flbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:12:32.410803 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.410769 2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e673909-83a4-48bf-8e92-1ac1dc71bede-dshm\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:12:32.410803 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.410796 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7flbk\" (UniqueName: \"kubernetes.io/projected/7e673909-83a4-48bf-8e92-1ac1dc71bede-kube-api-access-7flbk\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:12:32.410803 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.410809 2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e673909-83a4-48bf-8e92-1ac1dc71bede-tls-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:12:32.551320 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.551286 2550 generic.go:356] "Generic (PLEG): container finished" podID="7e673909-83a4-48bf-8e92-1ac1dc71bede" containerID="19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c" exitCode=0 Jul 14 15:12:32.551526 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.551329 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" Jul 14 15:12:32.551526 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.551379 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" event={"ID":"7e673909-83a4-48bf-8e92-1ac1dc71bede","Type":"ContainerDied","Data":"19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c"} Jul 14 15:12:32.551526 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.551422 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb" event={"ID":"7e673909-83a4-48bf-8e92-1ac1dc71bede","Type":"ContainerDied","Data":"c4ce701036a7c610c51ad2098738f10ed8a30db9f44661a4f719a4de1fa0f852"} Jul 14 15:12:32.551526 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.551440 2550 scope.go:117] "RemoveContainer" containerID="19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c" Jul 14 15:12:32.561298 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.561277 2550 scope.go:117] "RemoveContainer" containerID="19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c" Jul 14 15:12:32.561616 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:12:32.561598 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c\": container with ID starting with 19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c not found: ID does not exist" containerID="19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c" Jul 14 15:12:32.561692 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.561625 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c"} err="failed to get container status \"19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c\": rpc error: code = NotFound desc = could not find container \"19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c\": container with ID starting with 19ba19eccf86efce25a80afaf3ef368efcf78a2bd59e1498bcc679200375963c not found: ID does not exist" Jul 14 15:12:32.577891 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.577855 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb"] Jul 14 15:12:32.580063 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:32.580035 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/auth-test-model-ebb1a4-kserve-57b87b56d7-cqfcb"] Jul 14 15:12:33.120500 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:12:33.120470 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e673909-83a4-48bf-8e92-1ac1dc71bede" path="/var/lib/kubelet/pods/7e673909-83a4-48bf-8e92-1ac1dc71bede/volumes" Jul 14 15:13:34.969821 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.969731 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn"] Jul 14 15:13:34.970298 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.970169 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e673909-83a4-48bf-8e92-1ac1dc71bede" containerName="main" Jul 14 15:13:34.970298 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.970182 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e673909-83a4-48bf-8e92-1ac1dc71bede" containerName="main" Jul 14 15:13:34.970298 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.970247 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e673909-83a4-48bf-8e92-1ac1dc71bede" containerName="main" Jul 14 15:13:34.973380 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.973359 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:34.977426 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.977392 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ebb1a4\"/\"openshift-service-ca.crt\"" Jul 14 15:13:34.977426 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.977411 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ebb1a4\"/\"rate-test-model-ebb1a4-kserve-self-signed-certs\"" Jul 14 15:13:34.977675 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.977431 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ebb1a4\"/\"kube-root-ca.crt\"" Jul 14 15:13:34.977675 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.977461 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ebb1a4\"/\"default-dockercfg-dldgp\"" Jul 14 15:13:34.990115 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:34.990084 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn"] Jul 14 15:13:35.014065 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.014019 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-dshm\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.014268 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.014083 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tls-certs\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.014268 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.014132 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-home\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.014268 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.014164 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tmp-dir\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.014268 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.014195 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49q67\" (UniqueName: \"kubernetes.io/projected/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-kube-api-access-49q67\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.014268 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.014218 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-model-cache\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.114978 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.114940 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-dshm\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.115175 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.115015 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tls-certs\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.115175 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.115075 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-home\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.115175 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.115107 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tmp-dir\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.115175 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.115138 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q67\" (UniqueName: \"kubernetes.io/projected/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-kube-api-access-49q67\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.115175 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.115165 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-model-cache\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.115573 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.115540 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-model-cache\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.115758 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.115732 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-home\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.115944 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.115803 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tmp-dir\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.117592 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.117565 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-dshm\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.118122 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.118098 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tls-certs\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.129806 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.129767 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49q67\" (UniqueName: \"kubernetes.io/projected/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-kube-api-access-49q67\") pod \"rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.282796 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.282762 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:35.427878 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.427848 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn"] Jul 14 15:13:35.430717 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:13:35.430687 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f68f553_ba81_4c7e_b3a8_ad0c7cfe9e7e.slice/crio-dcd05245442ef814d7ec715fd22083e69bb5d8201fb31570523b28b89f59d0db WatchSource:0}: Error finding container dcd05245442ef814d7ec715fd22083e69bb5d8201fb31570523b28b89f59d0db: Status 404 returned error can't find the container with id dcd05245442ef814d7ec715fd22083e69bb5d8201fb31570523b28b89f59d0db Jul 14 15:13:35.880836 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.880800 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" event={"ID":"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e","Type":"ContainerStarted","Data":"f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4"} Jul 14 15:13:35.880836 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.880844 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" event={"ID":"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e","Type":"ContainerStarted","Data":"dcd05245442ef814d7ec715fd22083e69bb5d8201fb31570523b28b89f59d0db"} Jul 14 15:13:35.903098 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:35.903040 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" podStartSLOduration=1.903021844 podStartE2EDuration="1.903021844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:13:35.901501884 +0000 UTC m=+2605.336486508" watchObservedRunningTime="2026-07-14 15:13:35.903021844 +0000 UTC m=+2605.338006467" Jul 14 15:13:45.283561 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:45.283521 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:45.284190 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:45.283582 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:45.296146 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:45.296117 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:13:45.944703 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:13:45.944677 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:14:39.250806 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:39.250765 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn"] Jul 14 15:14:54.271230 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.271180 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" podUID="6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" containerName="main" containerID="cri-o://f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4" gracePeriod=45 Jul 14 15:14:54.525612 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.525547 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:14:54.585290 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585260 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-dshm\") pod \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " Jul 14 15:14:54.585492 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585333 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-model-cache\") pod \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " Jul 14 15:14:54.585492 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585361 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tmp-dir\") pod \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " Jul 14 15:14:54.585492 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585385 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49q67\" (UniqueName: \"kubernetes.io/projected/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-kube-api-access-49q67\") pod \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " Jul 14 15:14:54.585492 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585410 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-home\") pod \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " Jul 14 15:14:54.585492 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585450 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tls-certs\") pod \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\" (UID: \"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e\") " Jul 14 15:14:54.585735 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585633 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" (UID: "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:14:54.585735 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585721 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-home" (OuterVolumeSpecName: "home") pod "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" (UID: "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:14:54.585821 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585641 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" (UID: "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:14:54.585821 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.585804 2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tmp-dir\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:14:54.587502 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.587468 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" (UID: "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 15:14:54.587610 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.587513 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-kube-api-access-49q67" (OuterVolumeSpecName: "kube-api-access-49q67") pod "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" (UID: "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e"). InnerVolumeSpecName "kube-api-access-49q67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:14:54.587610 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.587548 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-dshm" (OuterVolumeSpecName: "dshm") pod "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" (UID: "6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:14:54.663844 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.663809 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:14:54.664282 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.664264 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" containerName="main" Jul 14 15:14:54.664282 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.664283 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" containerName="main" Jul 14 15:14:54.664438 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.664379 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" containerName="main" Jul 14 15:14:54.667379 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.667359 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:54.670558 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.670416 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-84wgl\"" Jul 14 15:14:54.673238 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.673215 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:14:54.686723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.686690 2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-tls-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:14:54.686723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.686713 2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-dshm\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:14:54.686723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.686722 2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-model-cache\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:14:54.686999 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.686731 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49q67\" (UniqueName: \"kubernetes.io/projected/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-kube-api-access-49q67\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:14:54.686999 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.686740 2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e-home\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:14:54.787280 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.787188 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhct6\" (UniqueName: \"kubernetes.io/projected/0f50d5b6-b27d-4abd-8f29-97510b44c829-kube-api-access-vhct6\") pod \"test-curl-7489-140672879944736\" (UID: \"0f50d5b6-b27d-4abd-8f29-97510b44c829\") " pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:54.888224 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.888189 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhct6\" (UniqueName: \"kubernetes.io/projected/0f50d5b6-b27d-4abd-8f29-97510b44c829-kube-api-access-vhct6\") pod \"test-curl-7489-140672879944736\" (UID: \"0f50d5b6-b27d-4abd-8f29-97510b44c829\") " pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:54.898295 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.898267 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhct6\" (UniqueName: \"kubernetes.io/projected/0f50d5b6-b27d-4abd-8f29-97510b44c829-kube-api-access-vhct6\") pod \"test-curl-7489-140672879944736\" (UID: \"0f50d5b6-b27d-4abd-8f29-97510b44c829\") " pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:54.977287 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:54.977254 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:55.111078 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.110925 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:14:55.113454 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:14:55.113420 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f50d5b6_b27d_4abd_8f29_97510b44c829.slice/crio-42f3400a9e0694f7b46903ef431300e98f0d2758b3909bf7f746ebf60e687c0b WatchSource:0}: Error finding container 42f3400a9e0694f7b46903ef431300e98f0d2758b3909bf7f746ebf60e687c0b: Status 404 returned error can't find the container with id 42f3400a9e0694f7b46903ef431300e98f0d2758b3909bf7f746ebf60e687c0b Jul 14 15:14:55.114723 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.114704 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 15:14:55.284368 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.284330 2550 generic.go:356] "Generic (PLEG): container finished" podID="6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" containerID="f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4" exitCode=0 Jul 14 15:14:55.284825 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.284375 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" Jul 14 15:14:55.284825 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.284411 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" event={"ID":"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e","Type":"ContainerDied","Data":"f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4"} Jul 14 15:14:55.284825 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.284456 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn" event={"ID":"6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e","Type":"ContainerDied","Data":"dcd05245442ef814d7ec715fd22083e69bb5d8201fb31570523b28b89f59d0db"} Jul 14 15:14:55.284825 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.284479 2550 scope.go:117] "RemoveContainer" containerID="f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4" Jul 14 15:14:55.285945 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.285921 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879944736" event={"ID":"0f50d5b6-b27d-4abd-8f29-97510b44c829","Type":"ContainerStarted","Data":"42f3400a9e0694f7b46903ef431300e98f0d2758b3909bf7f746ebf60e687c0b"} Jul 14 15:14:55.296045 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.296022 2550 scope.go:117] "RemoveContainer" containerID="f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4" Jul 14 15:14:55.296344 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:14:55.296324 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4\": container with ID starting with f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4 not found: ID does not exist" containerID="f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4" Jul 14 15:14:55.296396 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.296354 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4"} err="failed to get container status \"f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4\": rpc error: code = NotFound desc = could not find container \"f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4\": container with ID starting with f08bf6067a928202bedd8c0a72df17da6f8ae298592ca3c15c78a86dcba7f3b4 not found: ID does not exist" Jul 14 15:14:55.307296 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.307274 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn"] Jul 14 15:14:55.311316 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:55.311259 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ebb1a4/rate-test-model-ebb1a4-kserve-667bc58bcf-d8mzn"] Jul 14 15:14:56.294546 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:56.294508 2550 generic.go:356] "Generic (PLEG): container finished" podID="0f50d5b6-b27d-4abd-8f29-97510b44c829" containerID="f2b316808fb99675ac4337a2938969b676da25cc2e9a5ca4f020652680ca69cd" exitCode=0 Jul 14 15:14:56.294940 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:56.294585 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879944736" event={"ID":"0f50d5b6-b27d-4abd-8f29-97510b44c829","Type":"ContainerDied","Data":"f2b316808fb99675ac4337a2938969b676da25cc2e9a5ca4f020652680ca69cd"} Jul 14 15:14:57.121037 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.121003 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e" path="/var/lib/kubelet/pods/6f68f553-ba81-4c7e-b3a8-ad0c7cfe9e7e/volumes" Jul 14 15:14:57.449415 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.449385 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:57.493561 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.493536 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7489-140672879944736_0f50d5b6-b27d-4abd-8f29-97510b44c829/test-curl-7489-140672879944736/0.log" Jul 14 15:14:57.517819 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.517793 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhct6\" (UniqueName: \"kubernetes.io/projected/0f50d5b6-b27d-4abd-8f29-97510b44c829-kube-api-access-vhct6\") pod \"0f50d5b6-b27d-4abd-8f29-97510b44c829\" (UID: \"0f50d5b6-b27d-4abd-8f29-97510b44c829\") " Jul 14 15:14:57.519834 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.519808 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f50d5b6-b27d-4abd-8f29-97510b44c829-kube-api-access-vhct6" (OuterVolumeSpecName: "kube-api-access-vhct6") pod "0f50d5b6-b27d-4abd-8f29-97510b44c829" (UID: "0f50d5b6-b27d-4abd-8f29-97510b44c829"). InnerVolumeSpecName "kube-api-access-vhct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:14:57.530631 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.530604 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:14:57.533913 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.533890 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:14:57.618942 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.618906 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhct6\" (UniqueName: \"kubernetes.io/projected/0f50d5b6-b27d-4abd-8f29-97510b44c829-kube-api-access-vhct6\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:14:57.755271 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.755196 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:14:57.755598 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.755584 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f50d5b6-b27d-4abd-8f29-97510b44c829" containerName="test-curl-7489-140672879944736" Jul 14 15:14:57.755642 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.755599 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f50d5b6-b27d-4abd-8f29-97510b44c829" containerName="test-curl-7489-140672879944736" Jul 14 15:14:57.755688 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.755660 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f50d5b6-b27d-4abd-8f29-97510b44c829" containerName="test-curl-7489-140672879944736" Jul 14 15:14:57.758586 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.758570 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:57.780888 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.780862 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:14:57.821403 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.821363 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpz7w\" (UniqueName: \"kubernetes.io/projected/bb1f4ac0-486b-4340-a2f6-21f59b69b344-kube-api-access-qpz7w\") pod \"test-curl-7489-140672879944736\" (UID: \"bb1f4ac0-486b-4340-a2f6-21f59b69b344\") " pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:57.922225 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.922190 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpz7w\" (UniqueName: \"kubernetes.io/projected/bb1f4ac0-486b-4340-a2f6-21f59b69b344-kube-api-access-qpz7w\") pod \"test-curl-7489-140672879944736\" (UID: \"bb1f4ac0-486b-4340-a2f6-21f59b69b344\") " pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:57.931382 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:57.931348 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpz7w\" (UniqueName: \"kubernetes.io/projected/bb1f4ac0-486b-4340-a2f6-21f59b69b344-kube-api-access-qpz7w\") pod \"test-curl-7489-140672879944736\" (UID: \"bb1f4ac0-486b-4340-a2f6-21f59b69b344\") " pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:58.067666 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:58.067630 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:58.202951 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:58.202926 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:14:58.205309 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:14:58.205285 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb1f4ac0_486b_4340_a2f6_21f59b69b344.slice/crio-1feb02406dd2711b0f01e054197f6c71161ed4bfb8453f507177ce7e256da40f WatchSource:0}: Error finding container 1feb02406dd2711b0f01e054197f6c71161ed4bfb8453f507177ce7e256da40f: Status 404 returned error can't find the container with id 1feb02406dd2711b0f01e054197f6c71161ed4bfb8453f507177ce7e256da40f Jul 14 15:14:58.307253 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:58.307218 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879944736" event={"ID":"bb1f4ac0-486b-4340-a2f6-21f59b69b344","Type":"ContainerStarted","Data":"1feb02406dd2711b0f01e054197f6c71161ed4bfb8453f507177ce7e256da40f"} Jul 14 15:14:58.308434 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:58.308416 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:14:58.308548 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:58.308457 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f3400a9e0694f7b46903ef431300e98f0d2758b3909bf7f746ebf60e687c0b" Jul 14 15:14:58.321018 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:58.320934 2550 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7489-140672879944736" oldPodUID="0f50d5b6-b27d-4abd-8f29-97510b44c829" podUID="bb1f4ac0-486b-4340-a2f6-21f59b69b344" Jul 14 15:14:59.120762 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:59.120728 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f50d5b6-b27d-4abd-8f29-97510b44c829" path="/var/lib/kubelet/pods/0f50d5b6-b27d-4abd-8f29-97510b44c829/volumes" Jul 14 15:14:59.319706 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:59.319662 2550 generic.go:356] "Generic (PLEG): container finished" podID="bb1f4ac0-486b-4340-a2f6-21f59b69b344" containerID="411ba4170ccf8f98d31ff284a7c327664930cdaece51930dffcce51d67a70596" exitCode=0 Jul 14 15:14:59.319913 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:14:59.319740 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879944736" event={"ID":"bb1f4ac0-486b-4340-a2f6-21f59b69b344","Type":"ContainerDied","Data":"411ba4170ccf8f98d31ff284a7c327664930cdaece51930dffcce51d67a70596"} Jul 14 15:15:00.474152 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:00.474127 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:15:00.501151 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:00.501124 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7489-140672879944736_bb1f4ac0-486b-4340-a2f6-21f59b69b344/test-curl-7489-140672879944736/0.log" Jul 14 15:15:00.540838 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:00.540798 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:15:00.546906 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:00.546875 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7489-140672879944736"] Jul 14 15:15:00.548234 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:00.548215 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpz7w\" (UniqueName: \"kubernetes.io/projected/bb1f4ac0-486b-4340-a2f6-21f59b69b344-kube-api-access-qpz7w\") pod \"bb1f4ac0-486b-4340-a2f6-21f59b69b344\" (UID: \"bb1f4ac0-486b-4340-a2f6-21f59b69b344\") " Jul 14 15:15:00.550325 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:00.550290 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1f4ac0-486b-4340-a2f6-21f59b69b344-kube-api-access-qpz7w" (OuterVolumeSpecName: "kube-api-access-qpz7w") pod "bb1f4ac0-486b-4340-a2f6-21f59b69b344" (UID: "bb1f4ac0-486b-4340-a2f6-21f59b69b344"). InnerVolumeSpecName "kube-api-access-qpz7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:15:00.649685 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:00.649598 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpz7w\" (UniqueName: \"kubernetes.io/projected/bb1f4ac0-486b-4340-a2f6-21f59b69b344-kube-api-access-qpz7w\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:01.121308 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:01.121266 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb1f4ac0-486b-4340-a2f6-21f59b69b344" path="/var/lib/kubelet/pods/bb1f4ac0-486b-4340-a2f6-21f59b69b344/volumes" Jul 14 15:15:01.331539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:01.331508 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879944736" Jul 14 15:15:01.331539 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:01.331528 2550 scope.go:117] "RemoveContainer" containerID="411ba4170ccf8f98d31ff284a7c327664930cdaece51930dffcce51d67a70596" Jul 14 15:15:11.821367 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:11.821278 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 15:15:11.829296 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:11.829269 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6998d_09b8e867-76c5-4c35-b287-b2a2a1ee17a5/ovn-acl-logging/0.log" Jul 14 15:15:26.700634 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:26.700590 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc"] Jul 14 15:15:26.701107 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:26.700906 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" containerID="cri-o://d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8" gracePeriod=30 Jul 14 15:15:29.845757 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:29.845714 2550 patch_prober.go:28] interesting pod/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 15:15:29.846194 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:29.845775 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 15:15:31.959510 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:31.959481 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:15:32.041393 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041357 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-envoy\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041586 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041414 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-credential-socket\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041586 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041437 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-socket\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041586 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041453 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-certs\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041586 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041471 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx2qm\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-kube-api-access-vx2qm\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041586 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041548 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-data\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041586 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041574 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-podinfo\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041889 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041607 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a624859-5c7e-4b21-a263-7ca7ac650f40-istiod-ca-cert\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041889 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041629 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-token\") pod \"3a624859-5c7e-4b21-a263-7ca7ac650f40\" (UID: \"3a624859-5c7e-4b21-a263-7ca7ac650f40\") " Jul 14 15:15:32.041889 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041740 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:32.041889 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041845 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:32.041889 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041865 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:32.042153 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.041926 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.042212 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.042175 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a624859-5c7e-4b21-a263-7ca7ac650f40-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:15:32.042616 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.042589 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:32.043775 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.043747 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:32.043917 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.043894 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:15:32.044158 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.044132 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-kube-api-access-vx2qm" (OuterVolumeSpecName: "kube-api-access-vx2qm") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "kube-api-access-vx2qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:15:32.044259 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.044166 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a624859-5c7e-4b21-a263-7ca7ac650f40" (UID: "3a624859-5c7e-4b21-a263-7ca7ac650f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:15:32.142934 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.142892 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.142934 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.142925 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.142934 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.142935 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2qm\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-kube-api-access-vx2qm\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.142934 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.142944 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.143296 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.142953 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.143296 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.142977 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a624859-5c7e-4b21-a263-7ca7ac650f40-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.143296 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.142990 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.143296 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.143002 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a624859-5c7e-4b21-a263-7ca7ac650f40-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:32.487606 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.487514 2550 generic.go:356] "Generic (PLEG): container finished" podID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerID="d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8" exitCode=0 Jul 14 15:15:32.487606 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.487566 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" Jul 14 15:15:32.487788 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.487602 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" event={"ID":"3a624859-5c7e-4b21-a263-7ca7ac650f40","Type":"ContainerDied","Data":"d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8"} Jul 14 15:15:32.487788 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.487659 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc" event={"ID":"3a624859-5c7e-4b21-a263-7ca7ac650f40","Type":"ContainerDied","Data":"f6a99e26a38d6d1370ac265a7c309a971ec1dc87834529f58d7a26247af8e842"} Jul 14 15:15:32.487788 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.487683 2550 scope.go:117] "RemoveContainer" containerID="d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8" Jul 14 15:15:32.497012 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.496982 2550 scope.go:117] "RemoveContainer" containerID="f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177" Jul 14 15:15:32.505249 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.505221 2550 scope.go:117] "RemoveContainer" containerID="d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8" Jul 14 15:15:32.505590 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:15:32.505558 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8\": container with ID starting with d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8 not found: ID does not exist" containerID="d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8" Jul 14 15:15:32.505684 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.505601 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8"} err="failed to get container status \"d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8\": rpc error: code = NotFound desc = could not find container \"d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8\": container with ID starting with d8901bffc50a664554f1cb7376ee74739a5680a3edf2a29e337ab1831585d2f8 not found: ID does not exist" Jul 14 15:15:32.505684 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.505629 2550 scope.go:117] "RemoveContainer" containerID="f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177" Jul 14 15:15:32.505932 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:15:32.505910 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177\": container with ID starting with f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177 not found: ID does not exist" containerID="f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177" Jul 14 15:15:32.506016 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.505938 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177"} err="failed to get container status \"f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177\": rpc error: code = NotFound desc = could not find container \"f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177\": container with ID starting with f2a1b013521860238b986698de428b86a9c36b643db1dec46ae098351eec2177 not found: ID does not exist" Jul 14 15:15:32.513205 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.513178 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc"] Jul 14 15:15:32.522345 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:32.522320 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ebb1a4-openshift-default-5ff8777f4c-kthbc"] Jul 14 15:15:33.121192 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:33.121158 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" path="/var/lib/kubelet/pods/3a624859-5c7e-4b21-a263-7ca7ac650f40/volumes" Jul 14 15:15:52.958185 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:52.958146 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974"] Jul 14 15:15:52.959166 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:52.959128 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" containerID="cri-o://7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c" gracePeriod=30 Jul 14 15:15:53.043470 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:53.043432 2550 patch_prober.go:28] interesting pod/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 15:15:53.043649 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:53.043504 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 15:15:58.226139 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.226074 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:15:58.393183 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393151 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-certs\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393479 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393205 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-envoy\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393479 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393223 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-credential-socket\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393479 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393241 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-istio-token\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393479 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393260 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c5b997-f156-4b67-8495-295893791b9e-istiod-ca-cert\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393479 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393297 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c5b997-f156-4b67-8495-295893791b9e-istio-podinfo\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393479 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393330 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-data\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393479 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393366 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncjg5\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-kube-api-access-ncjg5\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393479 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393424 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-socket\") pod \"e1c5b997-f156-4b67-8495-295893791b9e\" (UID: \"e1c5b997-f156-4b67-8495-295893791b9e\") " Jul 14 15:15:58.393867 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393512 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:58.393867 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393536 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:58.393867 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393731 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:58.393867 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393806 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c5b997-f156-4b67-8495-295893791b9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 15:15:58.393867 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393826 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.393867 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393846 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-workload-certs\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.393867 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.393861 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-credential-socket\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.394472 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.394444 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:58.395483 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.395456 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:15:58.395580 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.395495 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 15:15:58.396052 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.396030 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1c5b997-f156-4b67-8495-295893791b9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 15:15:58.396195 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.396065 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-kube-api-access-ncjg5" (OuterVolumeSpecName: "kube-api-access-ncjg5") pod "e1c5b997-f156-4b67-8495-295893791b9e" (UID: "e1c5b997-f156-4b67-8495-295893791b9e"). InnerVolumeSpecName "kube-api-access-ncjg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 15:15:58.494716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.494611 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-envoy\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.494716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.494647 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-istio-token\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.494716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.494658 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c5b997-f156-4b67-8495-295893791b9e-istiod-ca-cert\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.494716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.494670 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c5b997-f156-4b67-8495-295893791b9e-istio-podinfo\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.494716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.494681 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c5b997-f156-4b67-8495-295893791b9e-istio-data\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.494716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.494693 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncjg5\" (UniqueName: \"kubernetes.io/projected/e1c5b997-f156-4b67-8495-295893791b9e-kube-api-access-ncjg5\") on node \"ip-10-0-141-194.ec2.internal\" DevicePath \"\"" Jul 14 15:15:58.624401 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.624366 2550 generic.go:356] "Generic (PLEG): container finished" podID="e1c5b997-f156-4b67-8495-295893791b9e" containerID="7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c" exitCode=0 Jul 14 15:15:58.624578 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.624420 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" Jul 14 15:15:58.624578 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.624454 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" event={"ID":"e1c5b997-f156-4b67-8495-295893791b9e","Type":"ContainerDied","Data":"7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c"} Jul 14 15:15:58.624578 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.624541 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974" event={"ID":"e1c5b997-f156-4b67-8495-295893791b9e","Type":"ContainerDied","Data":"cc9576c8353ee591a348f594c734734816bc094057fd286047380cd6bed3cd91"} Jul 14 15:15:58.624578 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.624568 2550 scope.go:117] "RemoveContainer" containerID="7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c" Jul 14 15:15:58.633622 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.633605 2550 scope.go:117] "RemoveContainer" containerID="7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36" Jul 14 15:15:58.641505 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.641481 2550 scope.go:117] "RemoveContainer" containerID="7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c" Jul 14 15:15:58.641752 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:15:58.641737 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c\": container with ID starting with 7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c not found: ID does not exist" containerID="7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c" Jul 14 15:15:58.641801 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.641760 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c"} err="failed to get container status \"7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c\": rpc error: code = NotFound desc = could not find container \"7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c\": container with ID starting with 7d02ea7c50500af7b56ab35ebdb641622b3748f51a2f2fba7e0a7acae22cf09c not found: ID does not exist" Jul 14 15:15:58.641801 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.641776 2550 scope.go:117] "RemoveContainer" containerID="7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36" Jul 14 15:15:58.641981 ip-10-0-141-194 kubenswrapper[2550]: E0714 15:15:58.641952 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36\": container with ID starting with 7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36 not found: ID does not exist" containerID="7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36" Jul 14 15:15:58.642026 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.641985 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36"} err="failed to get container status \"7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36\": rpc error: code = NotFound desc = could not find container \"7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36\": container with ID starting with 7f23d8b3d157df8cb9f1b122a56a5e8ac96b0e9ed44ca10a6a180f613b859b36 not found: ID does not exist" Jul 14 15:15:58.648631 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.648606 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974"] Jul 14 15:15:58.653462 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:58.653438 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-747ac7-openshift-default-794fb579f9-bz974"] Jul 14 15:15:59.120115 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:59.120087 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1c5b997-f156-4b67-8495-295893791b9e" path="/var/lib/kubelet/pods/e1c5b997-f156-4b67-8495-295893791b9e/volumes" Jul 14 15:15:59.202362 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:59.202333 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-42jgk_b5382ae7-ea4a-44df-bb08-9f2f28e8fea5/manager/0.log" Jul 14 15:15:59.339113 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:59.339072 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-g92jr_b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd/manager/0.log" Jul 14 15:15:59.728092 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:15:59.728058 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z8k4j_19d99917-a23a-46ed-8510-33748cdba09d/manager/0.log" Jul 14 15:16:00.099605 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:00.099580 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4469fb8c-mwtst_3caef96a-f494-4379-bd7e-0d46ec936597/copy-manifests/0.log" Jul 14 15:16:00.107053 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:00.107026 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4469fb8c-mwtst_3caef96a-f494-4379-bd7e-0d46ec936597/manager/0.log" Jul 14 15:16:02.514327 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:02.514295 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-bdwlt_060c90ba-b8e2-4746-a0f4-bf814683fbdf/postgres/0.log" Jul 14 15:16:04.120032 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:04.119996 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-4z7mc_47964024-b4b4-4e67-bc2a-635288b6b7af/manager/0.log" Jul 14 15:16:04.236084 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:04.236055 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s4f7r_768b3a77-486e-480c-94f7-0722aa4f35c5/manager/0.log" Jul 14 15:16:04.360648 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:04.360616 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-s7rvh_7e5222cd-dbf1-49ef-9a5d-fd9a347b609e/kuadrant-console-plugin/0.log" Jul 14 15:16:04.509133 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:04.509102 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xls6f_c9d3bd27-5de6-4be6-a06c-648ecab60c27/registry-server/0.log" Jul 14 15:16:04.918793 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:04.918702 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rq5l9_681b4a73-84d5-4c51-9c70-7608697f0ce4/manager/0.log" Jul 14 15:16:05.302716 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:05.302681 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447drmgjq_413f0850-488d-484b-ba18-36d661a1aec8/istio-proxy/0.log" Jul 14 15:16:05.573649 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:05.573565 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f65956896-6hpnv_4639828a-55fc-4f29-934f-fcf82a5e6b39/kube-auth-proxy/0.log" Jul 14 15:16:05.844680 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:05.844601 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-wvnd2_252f0ccf-34b0-4ab7-9641-c1b973022da1/istio-proxy/1.log" Jul 14 15:16:05.992587 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:05.992553 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-sq8qt_5b44233b-8db8-44b6-ad63-c360f042daf0/payload-pre-processing/0.log" Jul 14 15:16:06.124456 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:06.124381 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-xbmhh_15bc9e76-9962-4bd8-a3e5-c00926e5b794/payload-processing/0.log" Jul 14 15:16:06.769469 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:06.769422 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zmxg7_198fc64f-c361-40f7-92e7-090392167fa5/main/0.log" Jul 14 15:16:07.029744 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:07.029655 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbq7gqh_c62386e8-230b-45e2-8160-246390b8746c/main/0.log" Jul 14 15:16:07.311181 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:07.311089 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-tgdnz_7d21bbcb-60b4-4083-b6c9-87265f16814a/main/0.log" Jul 14 15:16:14.826014 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:14.825983 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wrmgv_ec9d1d25-83d7-4926-abe8-191d4e3fefe7/global-pull-secret-syncer/0.log" Jul 14 15:16:14.974224 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:14.974182 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ppp5w_768fa2c0-ba21-4175-b2bc-413f2080040d/konnectivity-agent/0.log" Jul 14 15:16:15.104070 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:15.103957 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-194.ec2.internal_1acd846e89bc70267fc722d6edc8fc43/haproxy/0.log" Jul 14 15:16:20.664426 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:20.664391 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-4z7mc_47964024-b4b4-4e67-bc2a-635288b6b7af/manager/0.log" Jul 14 15:16:20.695540 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:20.695511 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s4f7r_768b3a77-486e-480c-94f7-0722aa4f35c5/manager/0.log" Jul 14 15:16:20.726904 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:20.726878 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-s7rvh_7e5222cd-dbf1-49ef-9a5d-fd9a347b609e/kuadrant-console-plugin/0.log" Jul 14 15:16:20.780564 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:20.780527 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xls6f_c9d3bd27-5de6-4be6-a06c-648ecab60c27/registry-server/0.log" Jul 14 15:16:21.177806 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:21.177769 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rq5l9_681b4a73-84d5-4c51-9c70-7608697f0ce4/manager/0.log" Jul 14 15:16:23.199293 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:23.199261 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-kpqp2_0100d8ad-e992-42e7-acad-5fb3743c8c5c/monitoring-plugin/0.log" Jul 14 15:16:23.348889 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:23.348857 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74mth_0dec370b-da14-40f6-8ffe-6c05d5321e26/node-exporter/0.log" Jul 14 15:16:23.371111 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:23.371083 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74mth_0dec370b-da14-40f6-8ffe-6c05d5321e26/kube-rbac-proxy/0.log" Jul 14 15:16:23.392980 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:23.392940 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74mth_0dec370b-da14-40f6-8ffe-6c05d5321e26/init-textfile/0.log" Jul 14 15:16:25.189311 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189275 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq"] Jul 14 15:16:25.189829 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189808 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" Jul 14 15:16:25.189910 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189831 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" Jul 14 15:16:25.189910 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189842 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" Jul 14 15:16:25.189910 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189850 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" Jul 14 15:16:25.189910 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189881 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" Jul 14 15:16:25.189910 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189889 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" Jul 14 15:16:25.189910 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189909 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1f4ac0-486b-4340-a2f6-21f59b69b344" containerName="test-curl-7489-140672879944736" Jul 14 15:16:25.190219 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189921 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1f4ac0-486b-4340-a2f6-21f59b69b344" containerName="test-curl-7489-140672879944736" Jul 14 15:16:25.190219 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189935 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" Jul 14 15:16:25.190219 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.189944 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" Jul 14 15:16:25.190219 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.190064 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a624859-5c7e-4b21-a263-7ca7ac650f40" containerName="istio-proxy" Jul 14 15:16:25.190219 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.190077 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1c5b997-f156-4b67-8495-295893791b9e" containerName="istio-proxy" Jul 14 15:16:25.190219 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.190092 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1f4ac0-486b-4340-a2f6-21f59b69b344" containerName="test-curl-7489-140672879944736" Jul 14 15:16:25.193153 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.193131 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.196159 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.196140 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x95g4\"/\"kube-root-ca.crt\"" Jul 14 15:16:25.197426 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.197407 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x95g4\"/\"default-dockercfg-czkgw\"" Jul 14 15:16:25.197531 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.197441 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x95g4\"/\"openshift-service-ca.crt\"" Jul 14 15:16:25.207365 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.207334 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq"] Jul 14 15:16:25.240381 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.240343 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-proc\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.240381 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.240385 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-lib-modules\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.240595 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.240407 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-sys\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.240595 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.240424 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpkb\" (UniqueName: \"kubernetes.io/projected/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-kube-api-access-zvpkb\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.240595 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.240461 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-podres\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.341858 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.341826 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-proc\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.342065 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.341861 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-lib-modules\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.342065 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.341888 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-sys\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.342065 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.341905 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpkb\" (UniqueName: \"kubernetes.io/projected/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-kube-api-access-zvpkb\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.342065 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.341936 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-podres\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.342065 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.341978 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-proc\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.342065 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.342000 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-sys\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.342065 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.342030 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-lib-modules\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.342332 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.342102 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-podres\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.352825 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.352793 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpkb\" (UniqueName: \"kubernetes.io/projected/1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6-kube-api-access-zvpkb\") pod \"perf-node-gather-daemonset-mmmrq\" (UID: \"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.503478 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.503379 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.643159 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.643128 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq"] Jul 14 15:16:25.646062 ip-10-0-141-194 kubenswrapper[2550]: W0714 15:16:25.646032 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1ead6682_f2b1_4d9e_a3e0_92e6bd23bda6.slice/crio-e7b8932df7864bb6a16cc16eef3ed4546177f7da8c6483da3c1bf0da2345f8c7 WatchSource:0}: Error finding container e7b8932df7864bb6a16cc16eef3ed4546177f7da8c6483da3c1bf0da2345f8c7: Status 404 returned error can't find the container with id e7b8932df7864bb6a16cc16eef3ed4546177f7da8c6483da3c1bf0da2345f8c7 Jul 14 15:16:25.756413 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.756383 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" event={"ID":"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6","Type":"ContainerStarted","Data":"975a628572448a12846b8803788131c3ee21d98e3e26001f708d51d97de7fc11"} Jul 14 15:16:25.756584 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.756443 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" event={"ID":"1ead6682-f2b1-4d9e-a3e0-92e6bd23bda6","Type":"ContainerStarted","Data":"e7b8932df7864bb6a16cc16eef3ed4546177f7da8c6483da3c1bf0da2345f8c7"} Jul 14 15:16:25.756584 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.756471 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:25.778100 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:25.778051 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" podStartSLOduration=0.778037375 podStartE2EDuration="778.037375ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:16:25.775754625 +0000 UTC m=+2775.210739248" watchObservedRunningTime="2026-07-14 15:16:25.778037375 +0000 UTC m=+2775.213021999" Jul 14 15:16:26.527601 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:26.527566 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74d88fc5fb-xrzss_c16f4571-c50e-46dc-890f-91706ec57415/console/0.log" Jul 14 15:16:26.603156 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:26.603125 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-6h6nq_d58b7a59-b913-445f-bec8-afe6c0fb75c3/download-server/0.log" Jul 14 15:16:27.198396 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:27.198305 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-w6pbn_2d287354-d90b-40e8-9b59-0a88c906081b/volume-data-source-validator/0.log" Jul 14 15:16:28.289816 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:28.289766 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzwj6_bc7a7851-c2e6-414a-9716-5296d9c8e416/dns/0.log" Jul 14 15:16:28.313410 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:28.313372 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzwj6_bc7a7851-c2e6-414a-9716-5296d9c8e416/kube-rbac-proxy/0.log" Jul 14 15:16:28.458201 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:28.458171 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w4pnw_c078bb29-1f4d-45f5-beed-4d91e9a4a288/dns-node-resolver/0.log" Jul 14 15:16:29.095325 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:29.095284 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mrt4j_6b2c2d3f-0746-4db3-9fd8-f486ecdaf82a/node-ca/0.log" Jul 14 15:16:30.086325 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:30.086294 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447drmgjq_413f0850-488d-484b-ba18-36d661a1aec8/istio-proxy/0.log" Jul 14 15:16:30.318297 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:30.318272 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f65956896-6hpnv_4639828a-55fc-4f29-934f-fcf82a5e6b39/kube-auth-proxy/0.log" Jul 14 15:16:30.378347 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:30.378259 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-wvnd2_252f0ccf-34b0-4ab7-9641-c1b973022da1/istio-proxy/0.log" Jul 14 15:16:30.437599 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:30.437564 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-wvnd2_252f0ccf-34b0-4ab7-9641-c1b973022da1/istio-proxy/1.log" Jul 14 15:16:30.478480 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:30.478448 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-sq8qt_5b44233b-8db8-44b6-ad63-c360f042daf0/payload-pre-processing/0.log" Jul 14 15:16:30.519269 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:30.519234 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-xbmhh_15bc9e76-9962-4bd8-a3e5-c00926e5b794/payload-processing/0.log" Jul 14 15:16:31.301052 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:31.301003 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tsdfc_d24875c9-5d09-428e-a559-03ea92fbb9dd/serve-healthcheck-canary/0.log" Jul 14 15:16:31.770647 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:31.770614 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-mmmrq" Jul 14 15:16:32.022533 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:32.022448 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xs4rs_743dbe3e-0b78-4729-9eea-58edd00051e3/kube-rbac-proxy/0.log" Jul 14 15:16:32.046315 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:32.046289 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xs4rs_743dbe3e-0b78-4729-9eea-58edd00051e3/exporter/0.log" Jul 14 15:16:32.070631 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:32.070606 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xs4rs_743dbe3e-0b78-4729-9eea-58edd00051e3/extractor/0.log" Jul 14 15:16:34.479890 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:34.479783 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-42jgk_b5382ae7-ea4a-44df-bb08-9f2f28e8fea5/manager/0.log" Jul 14 15:16:34.728066 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:34.728014 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-g92jr_b70536a7-1cfa-4a3e-a46e-d1ad6c468cbd/manager/0.log" Jul 14 15:16:35.040833 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.040752 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z8k4j_19d99917-a23a-46ed-8510-33748cdba09d/manager/0.log" Jul 14 15:16:35.261452 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.261422 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4469fb8c-mwtst_3caef96a-f494-4379-bd7e-0d46ec936597/manager/0.log" Jul 14 15:16:35.285543 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.285521 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4469fb8c-mwtst_3caef96a-f494-4379-bd7e-0d46ec936597/copy-manifests/0.log" Jul 14 15:16:35.596950 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.596910 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447drmgjq_413f0850-488d-484b-ba18-36d661a1aec8/istio-proxy/0.log" Jul 14 15:16:35.796649 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.796614 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f65956896-6hpnv_4639828a-55fc-4f29-934f-fcf82a5e6b39/kube-auth-proxy/0.log" Jul 14 15:16:35.839521 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.839486 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-wvnd2_252f0ccf-34b0-4ab7-9641-c1b973022da1/istio-proxy/0.log" Jul 14 15:16:35.907821 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.907727 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-wvnd2_252f0ccf-34b0-4ab7-9641-c1b973022da1/istio-proxy/1.log" Jul 14 15:16:35.936207 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.936164 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-sq8qt_5b44233b-8db8-44b6-ad63-c360f042daf0/payload-pre-processing/0.log" Jul 14 15:16:35.966736 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:35.966695 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-xbmhh_15bc9e76-9962-4bd8-a3e5-c00926e5b794/payload-processing/0.log" Jul 14 15:16:36.561930 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:36.561899 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-rsplw_2bb166fd-1d62-4ee7-b587-0343283cde3a/openshift-lws-operator/0.log" Jul 14 15:16:37.256910 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:37.256885 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447drmgjq_413f0850-488d-484b-ba18-36d661a1aec8/istio-proxy/0.log" Jul 14 15:16:37.495708 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:37.495671 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f65956896-6hpnv_4639828a-55fc-4f29-934f-fcf82a5e6b39/kube-auth-proxy/0.log" Jul 14 15:16:37.557026 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:37.556900 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-wvnd2_252f0ccf-34b0-4ab7-9641-c1b973022da1/istio-proxy/0.log" Jul 14 15:16:37.619194 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:37.619161 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-wvnd2_252f0ccf-34b0-4ab7-9641-c1b973022da1/istio-proxy/1.log" Jul 14 15:16:37.649674 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:37.649642 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-sq8qt_5b44233b-8db8-44b6-ad63-c360f042daf0/payload-pre-processing/0.log" Jul 14 15:16:37.681849 ip-10-0-141-194 kubenswrapper[2550]: I0714 15:16:37.681814 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-xbmhh_15bc9e76-9962-4bd8-a3e5-c00926e5b794/payload-processing/0.log"