Jul 16 17:28:56.937947 ip-10-0-136-229 systemd[1]: Starting Kubernetes Kubelet... Jul 16 17:28:57.269634 ip-10-0-136-229 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:28:57.269634 ip-10-0-136-229 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 17:28:57.269634 ip-10-0-136-229 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:28:57.269634 ip-10-0-136-229 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 17:28:57.269634 ip-10-0-136-229 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:28:57.270463 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.270368 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 17:28:57.275023 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275000 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:28:57.275023 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275023 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275028 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275031 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275034 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275037 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275040 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275043 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275047 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275049 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275052 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275055 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275058 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275061 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275064 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275066 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275069 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275074 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275078 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275080 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275084 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:28:57.275098 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275088 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275091 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275093 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275096 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275106 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275108 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275111 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275114 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275117 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275119 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275122 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275125 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275128 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275130 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275133 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275135 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275137 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275142 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275145 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:28:57.275563 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275150 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275153 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275156 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275159 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275161 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275164 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275166 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275169 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275171 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275174 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275177 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275179 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275181 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275184 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275187 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275191 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275194 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275196 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275199 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275201 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:28:57.276069 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275204 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275206 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275209 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275211 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275213 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275216 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275218 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275221 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275223 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275226 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275228 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275231 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275233 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275236 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275238 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275240 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275243 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275245 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275248 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275250 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:28:57.276551 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275253 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275255 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275258 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275260 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275263 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275265 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275268 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275271 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275274 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275277 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275279 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275282 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275284 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275287 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275289 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275292 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275294 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275296 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:28:57.277043 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.275300 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277273 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277282 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277285 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277288 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277292 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277295 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277297 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277300 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277304 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277306 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277309 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277312 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277314 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277317 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277319 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277322 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277330 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277333 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277335 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:28:57.277464 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277338 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277341 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277343 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277345 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277349 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277353 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277356 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277358 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277361 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277364 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277366 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277369 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277371 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277374 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277377 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277380 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277382 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277385 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277388 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277390 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:28:57.278055 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277393 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277395 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277398 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277400 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277402 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277405 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277408 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277410 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277412 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277415 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277423 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277426 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277429 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277431 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277433 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277436 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277438 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277442 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277445 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:28:57.278575 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277447 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277450 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277453 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277455 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277458 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277460 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277462 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277466 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277468 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277471 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277473 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277476 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277479 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277481 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277483 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277486 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277488 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277491 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277493 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277495 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:28:57.279051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277498 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277501 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277504 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277506 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277513 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277515 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277518 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277520 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277523 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277525 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277540 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277544 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277547 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277550 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277553 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277555 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277564 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277568 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277571 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277577 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:28:57.279541 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.277580 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278002 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278008 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278010 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278014 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278016 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278019 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278021 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278024 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278026 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278029 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278031 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278034 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278036 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278039 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278041 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278044 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278047 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278050 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278052 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:28:57.280022 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278055 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278057 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278060 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278063 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278065 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278068 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278070 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278073 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278075 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278078 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278080 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278092 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278095 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278098 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278100 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278103 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278105 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278108 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278110 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278114 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:28:57.280508 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278116 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278119 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278121 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278124 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278126 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278129 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278131 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278134 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278136 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278144 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278147 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278149 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278152 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278155 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278158 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278161 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278163 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278165 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278168 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:28:57.281010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278170 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278173 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278175 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278178 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278182 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278184 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278188 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278192 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278194 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278197 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278199 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278216 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278220 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278223 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278226 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278228 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278231 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278234 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278238 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:28:57.281463 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278241 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278244 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278247 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278249 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278258 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278260 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278263 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278265 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278268 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278270 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278273 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278275 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278277 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278280 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278282 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278284 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278287 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278291 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278293 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278296 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:28:57.281929 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278299 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.278301 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278373 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278381 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278387 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278394 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278400 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278405 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278411 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278418 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278423 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278426 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278430 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278433 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278436 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278439 2582 flags.go:64] FLAG: --cgroup-root="" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278442 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278446 2582 flags.go:64] FLAG: --client-ca-file="" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278449 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278452 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278456 2582 flags.go:64] FLAG: --cluster-domain="" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278459 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278462 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278465 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 17:28:57.282395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278468 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278472 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278475 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278478 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278481 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278486 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278489 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278492 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278495 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278498 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278502 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278505 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278508 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278511 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278514 2582 flags.go:64] FLAG: --enable-server="true" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278517 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278521 2582 flags.go:64] FLAG: --event-burst="100" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278524 2582 flags.go:64] FLAG: --event-qps="50" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278527 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278530 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278533 2582 flags.go:64] FLAG: --eviction-hard="" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278537 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278540 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278543 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278546 2582 flags.go:64] FLAG: --eviction-soft="" Jul 16 17:28:57.282972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278548 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278552 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278555 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278558 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278560 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278563 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278566 2582 flags.go:64] FLAG: --feature-gates="" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278570 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278573 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278576 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278579 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278582 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278588 2582 flags.go:64] FLAG: --help="false" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278590 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278594 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278614 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278617 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278620 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278624 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278627 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278630 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278632 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278635 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 17:28:57.283550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278638 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278641 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278644 2582 flags.go:64] FLAG: --kube-reserved="" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278647 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278649 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278652 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278655 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278658 2582 flags.go:64] FLAG: --lock-file="" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278661 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278663 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278666 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278672 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278676 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278678 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278681 2582 flags.go:64] FLAG: --logging-format="text" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278684 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278687 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278690 2582 flags.go:64] FLAG: --manifest-url="" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278693 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278697 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278700 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278706 2582 flags.go:64] FLAG: --max-pods="110" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278710 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278712 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278715 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 17:28:57.284177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278718 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278721 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278724 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278727 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278734 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278737 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278740 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278743 2582 flags.go:64] FLAG: --pod-cidr="" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278746 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278752 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278755 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278758 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278761 2582 flags.go:64] FLAG: --port="10250" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278764 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278767 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08b9bd1b31dadaed2" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278770 2582 flags.go:64] FLAG: --qos-reserved="" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278773 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278775 2582 flags.go:64] FLAG: --register-node="true" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278778 2582 flags.go:64] FLAG: --register-with-taints="" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278788 2582 flags.go:64] FLAG: --registry-burst="10" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278791 2582 flags.go:64] FLAG: --registry-qps="5" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278794 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278797 2582 flags.go:64] FLAG: --reserved-memory="" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278801 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278803 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 17:28:57.284788 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278806 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278809 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278812 2582 flags.go:64] FLAG: --runonce="false" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278816 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278820 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278824 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278826 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278829 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278832 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278835 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278838 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278841 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278844 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278846 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278849 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278852 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278854 2582 flags.go:64] FLAG: --system-cgroups="" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278857 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278862 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278865 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278868 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278872 2582 flags.go:64] FLAG: --tls-min-version="" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278875 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278877 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278880 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 17:28:57.285385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278883 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278886 2582 flags.go:64] FLAG: --v="2" Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278895 2582 flags.go:64] FLAG: --version="false" Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278899 2582 flags.go:64] FLAG: --vmodule="" Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278903 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.278906 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279034 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279038 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279041 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279043 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279047 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279051 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279055 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279058 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279061 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279064 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279066 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279069 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279071 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279074 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279076 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:28:57.285996 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279079 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279082 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279084 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279087 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279089 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279092 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279094 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279096 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279099 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279102 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279104 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279107 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279109 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279112 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279115 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279117 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279120 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279122 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279124 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279127 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:28:57.286501 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279129 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279132 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279137 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279139 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279142 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279144 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279146 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279149 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279151 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279154 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279156 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279159 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279161 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279164 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279166 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279169 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279171 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279175 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279178 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:28:57.287056 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279181 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279183 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279186 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279188 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279190 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279192 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279195 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279199 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279203 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279205 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279208 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279211 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279213 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279216 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279218 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279222 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279225 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279227 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279230 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:28:57.287840 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279233 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279236 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279238 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279241 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279244 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279246 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279249 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279251 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279253 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279256 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279258 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279261 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279263 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279266 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279268 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279270 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279273 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279276 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279278 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279281 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:28:57.288401 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279283 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279285 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279289 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279292 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279296 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.279298 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.279855 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.288228 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.288373 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.288403 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.288409 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288478 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288484 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288488 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:28:57.288902 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288491 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288494 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288497 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288500 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288503 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288506 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288509 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288511 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288514 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288516 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288519 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288521 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288524 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288526 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288529 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288532 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288535 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288539 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288541 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:28:57.289249 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288544 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288546 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288549 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288551 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288554 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288557 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288560 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288563 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288566 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288568 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288571 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288573 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288577 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288581 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288585 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288587 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288590 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288593 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288614 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288618 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:28:57.289720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288621 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288623 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288626 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288629 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288631 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288634 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288637 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288639 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288641 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288645 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288647 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288650 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288652 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288654 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288657 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288661 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288664 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288667 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288669 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:28:57.290205 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288673 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288676 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288679 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288682 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288685 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288688 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288691 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288693 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288696 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288699 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288701 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288705 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288708 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288710 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288713 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288715 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288718 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288720 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288723 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288725 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:28:57.290664 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288728 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288731 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288733 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288736 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288739 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288741 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288743 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288746 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288748 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288751 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288753 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288755 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288759 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288762 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288764 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288768 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288771 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:28:57.291178 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288775 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.288780 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288896 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288902 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288905 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288908 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288911 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288914 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288917 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288919 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288922 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288925 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288928 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288931 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:28:57.291579 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288934 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288937 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288940 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288943 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288945 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288947 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288950 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288952 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288955 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288958 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288960 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288963 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288965 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288968 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288971 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288974 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288976 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288979 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288982 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288984 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:28:57.291927 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288986 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288989 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288992 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.288995 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289000 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289002 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289005 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289007 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289010 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289012 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289014 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289017 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289019 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289025 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289028 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289030 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289033 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289036 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289038 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:28:57.292407 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289041 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289043 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289045 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289047 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289050 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289052 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289055 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289058 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289060 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289063 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289065 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289068 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289070 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289072 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289075 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289077 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289079 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289082 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289085 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:28:57.292876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289089 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289092 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289095 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289098 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289100 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289103 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289106 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289108 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289112 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289114 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289117 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289120 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289122 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289125 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289127 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289130 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289132 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289135 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289137 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289140 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:28:57.293324 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289142 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289145 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289147 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289150 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289152 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289155 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289157 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289161 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:57.289164 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.289169 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.289314 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.292462 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.293675 2582 server.go:1019] "Starting client certificate rotation" Jul 16 17:28:57.293817 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.293776 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:28:57.294138 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.293811 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:28:57.295821 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.295805 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:28:57.299633 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.299583 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:28:57.307754 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.307733 2582 log.go:25] "Validated CRI v1 runtime API" Jul 16 17:28:57.309789 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.309767 2582 log.go:25] "Validated CRI v1 image API" Jul 16 17:28:57.310642 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.310623 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 17:28:57.312803 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.312779 2582 fs.go:135] Filesystem UUIDs: map[273747f7-b823-426d-a1d8-cf70618a9b6c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b78cb62f-c99f-4bb8-afc9-ffb141d44dc8:/dev/nvme0n1p3] Jul 16 17:28:57.312866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.312804 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 17:28:57.319247 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.319105 2582 manager.go:216] Machine: {Timestamp:2026-07-16 17:28:57.317179459 +0000 UTC m=+0.272499838 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105401 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f851b26bc8bba8e15939e5caa59de SystemUUID:ec2f851b-26bc-8bba-8e15-939e5caa59de BootID:1d08db16-19fa-46a5-baaf-27a3ce3c1bd4 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:f3:3b:30:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:f3:3b:30:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:88:00:52:10:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 17:28:57.319247 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.319234 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 17:28:57.319379 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.319334 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 17:28:57.320844 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.320812 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 17:28:57.321003 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.320853 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 17:28:57.321526 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.321513 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 17:28:57.321564 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.321528 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 17:28:57.321564 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.321538 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:28:57.322102 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.322090 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:28:57.322814 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.322802 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 17:28:57.326538 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.326520 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:28:57.326685 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.326674 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 17:28:57.326727 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.326701 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 16 17:28:57.326727 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.326713 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 17:28:57.326727 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.326724 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 17:28:57.326845 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.326735 2582 kubelet.go:398] "Adding apiserver pod source" Jul 16 17:28:57.326845 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.326746 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 17:28:57.326912 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.326853 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:28:57.327728 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.327714 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:28:57.327784 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.327735 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:28:57.331475 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.331452 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 17:28:57.332792 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.332778 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 17:28:57.332843 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.332810 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 17:28:57.333425 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333413 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 17:28:57.333458 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333436 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 17:28:57.333458 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333446 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 17:28:57.333458 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333453 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 17:28:57.333532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333463 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 17:28:57.333532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333468 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 17:28:57.333532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333476 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 17:28:57.333532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333482 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 17:28:57.333532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333489 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 17:28:57.333532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333495 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 17:28:57.333532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333503 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 17:28:57.333532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.333508 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 17:28:57.334025 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.334015 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 17:28:57.334054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.334028 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 17:28:57.337639 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.337625 2582 server.go:1268] "Started kubelet" Jul 16 17:28:57.337776 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.337728 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 17:28:57.337835 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.337796 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 17:28:57.337835 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.337791 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 17:28:57.338558 ip-10-0-136-229 systemd[1]: Started Kubernetes Kubelet. Jul 16 17:28:57.338763 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.338746 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 17:28:57.339797 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.339781 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 16 17:28:57.341776 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.341756 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 17:28:57.342292 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.342257 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 17:28:57.342292 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.342266 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 17:28:57.343014 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.342165 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-229.ec2.internal.18c2d54eec1c483a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-229.ec2.internal,UID:ip-10-0-136-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-229.ec2.internal,},FirstTimestamp:2026-07-16 17:28:57.337579578 +0000 UTC m=+0.292899935,LastTimestamp:2026-07-16 17:28:57.337579578 +0000 UTC m=+0.292899935,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-229.ec2.internal,}" Jul 16 17:28:57.343284 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.343268 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 17:28:57.343390 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.343333 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 17:28:57.343446 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.343419 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:57.343803 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.343788 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 17:28:57.343939 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.343923 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 17:28:57.344025 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.343835 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 17:28:57.344025 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.343831 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 17:28:57.344100 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344060 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 16 17:28:57.344100 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344070 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 16 17:28:57.344360 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344346 2582 factory.go:55] Registering systemd factory Jul 16 17:28:57.344425 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344366 2582 factory.go:223] Registration of the systemd container factory successfully Jul 16 17:28:57.344687 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344663 2582 factory.go:153] Registering CRI-O factory Jul 16 17:28:57.344687 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344678 2582 factory.go:223] Registration of the crio container factory successfully Jul 16 17:28:57.344794 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344728 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 17:28:57.344794 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344752 2582 factory.go:103] Registering Raw factory Jul 16 17:28:57.344794 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.344775 2582 manager.go:1196] Started watching for new ooms in manager Jul 16 17:28:57.345128 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.345119 2582 manager.go:349] Starting recovery of all containers Jul 16 17:28:57.346176 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.346152 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 17:28:57.346264 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.346245 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 17:28:57.356706 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.354991 2582 manager.go:354] Recovery completed Jul 16 17:28:57.360524 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.360505 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:28:57.363367 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.363337 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:28:57.363448 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.363387 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:28:57.363448 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.363401 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:28:57.364287 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.364269 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 17:28:57.364287 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.364285 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 17:28:57.364391 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.364300 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:28:57.364391 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.364274 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:28:57.365070 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.365057 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:28:57.365124 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.365087 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:28:57.365124 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.365097 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:28:57.365678 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.365588 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-229.ec2.internal.18c2d54eeda5caa0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-229.ec2.internal,UID:ip-10-0-136-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-229.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-229.ec2.internal,},FirstTimestamp:2026-07-16 17:28:57.363368608 +0000 UTC m=+0.318688974,LastTimestamp:2026-07-16 17:28:57.363368608 +0000 UTC m=+0.318688974,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-229.ec2.internal,}" Jul 16 17:28:57.366683 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.366670 2582 policy_none.go:49] "None policy: Start" Jul 16 17:28:57.366739 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.366690 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 17:28:57.366739 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.366708 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 17:28:57.368345 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.368321 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p6x5c" Jul 16 17:28:57.368765 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.368750 2582 policy_none.go:47] "Start" Jul 16 17:28:57.375548 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.375476 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-229.ec2.internal.18c2d54eeda62c5f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-229.ec2.internal,UID:ip-10-0-136-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-229.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-229.ec2.internal,},FirstTimestamp:2026-07-16 17:28:57.363393631 +0000 UTC m=+0.318713988,LastTimestamp:2026-07-16 17:28:57.363393631 +0000 UTC m=+0.318713988,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-229.ec2.internal,}" Jul 16 17:28:57.377223 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.377207 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p6x5c" Jul 16 17:28:57.399776 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.399760 2582 manager.go:337] "Starting Device Plugin manager" Jul 16 17:28:57.399880 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.399796 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 17:28:57.399880 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.399810 2582 server.go:87] "Starting device plugin registration server" Jul 16 17:28:57.400113 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.400101 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 17:28:57.400182 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.400113 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 17:28:57.400258 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.400245 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 17:28:57.400373 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.400350 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 17:28:57.400373 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.400362 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 17:28:57.400886 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.400870 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 17:28:57.400955 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.400913 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:57.439559 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.439512 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 17:28:57.445852 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.441123 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 17:28:57.445852 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.441166 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 17:28:57.445852 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.441196 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 17:28:57.445852 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.441233 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 17:28:57.445852 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.444047 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:28:57.500285 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.500249 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:28:57.501413 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.501393 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:28:57.501523 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.501430 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:28:57.501523 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.501443 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:28:57.501523 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.501472 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.511540 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.511518 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.511646 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.511545 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-229.ec2.internal\": node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:57.533537 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.533471 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:57.542024 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.541992 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal"] Jul 16 17:28:57.542139 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.542082 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:28:57.543091 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.543072 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:28:57.543202 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.543106 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:28:57.543202 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.543120 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:28:57.545509 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.545494 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:28:57.545643 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.545628 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.545705 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.545665 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:28:57.546224 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.546206 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:28:57.546293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.546234 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:28:57.546293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.546236 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:28:57.546293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.546254 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:28:57.546293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.546262 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:28:57.546293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.546269 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:28:57.548553 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.548537 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.548641 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.548563 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:28:57.549338 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.549311 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:28:57.549390 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.549339 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:28:57.549390 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.549356 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:28:57.573556 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.573523 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-229.ec2.internal\" not found" node="ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.577095 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.577074 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-229.ec2.internal\" not found" node="ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.633636 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.633581 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:57.644791 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.644761 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8456db5991817cae9ae3a70301fb5f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal\" (UID: \"d8456db5991817cae9ae3a70301fb5f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.644922 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.644795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8456db5991817cae9ae3a70301fb5f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal\" (UID: \"d8456db5991817cae9ae3a70301fb5f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.734254 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.734217 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:57.745443 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.745410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8456db5991817cae9ae3a70301fb5f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal\" (UID: \"d8456db5991817cae9ae3a70301fb5f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.745592 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.745446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8456db5991817cae9ae3a70301fb5f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal\" (UID: \"d8456db5991817cae9ae3a70301fb5f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.745592 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.745467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b041e55b23cc7cb3930d7a715f578cc-config\") pod \"kube-apiserver-proxy-ip-10-0-136-229.ec2.internal\" (UID: \"0b041e55b23cc7cb3930d7a715f578cc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.745592 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.745511 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8456db5991817cae9ae3a70301fb5f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal\" (UID: \"d8456db5991817cae9ae3a70301fb5f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.745592 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.745530 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8456db5991817cae9ae3a70301fb5f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal\" (UID: \"d8456db5991817cae9ae3a70301fb5f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.834950 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.834875 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:57.846226 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.846200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b041e55b23cc7cb3930d7a715f578cc-config\") pod \"kube-apiserver-proxy-ip-10-0-136-229.ec2.internal\" (UID: \"0b041e55b23cc7cb3930d7a715f578cc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.846289 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.846264 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b041e55b23cc7cb3930d7a715f578cc-config\") pod \"kube-apiserver-proxy-ip-10-0-136-229.ec2.internal\" (UID: \"0b041e55b23cc7cb3930d7a715f578cc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.876383 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.876355 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.879912 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:57.879892 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" Jul 16 17:28:57.935926 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:57.935891 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:58.036440 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:58.036396 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:58.137056 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:58.136966 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:58.237540 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:58.237495 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:58.293085 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.293056 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 17:28:58.293713 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.293234 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 17:28:58.338677 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:58.338633 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:58.382269 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.382222 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:23:58 +0000 UTC" deadline="2028-03-25 06:37:40.1572947 +0000 UTC" Jul 16 17:28:58.382269 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.382279 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14821h8m41.775040694s" Jul 16 17:28:58.416849 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.416777 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:28:58.418901 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.418879 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:28:58.419039 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.418916 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:28:58.419039 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.418931 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:28:58.419806 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.419789 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:28:58.439647 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:58.439612 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:58.497749 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.497721 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:28:58.540014 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:58.539991 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:58.556729 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:28:58.556685 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8456db5991817cae9ae3a70301fb5f2.slice/crio-b211bdc927eab2c3991ed483e3801789d35b507375f0f3cdcf862da9abcd5da9 WatchSource:0}: Error finding container b211bdc927eab2c3991ed483e3801789d35b507375f0f3cdcf862da9abcd5da9: Status 404 returned error can't find the container with id b211bdc927eab2c3991ed483e3801789d35b507375f0f3cdcf862da9abcd5da9 Jul 16 17:28:58.560766 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:58.560751 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:28:59.844244 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:59.844205 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:59.944815 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:28:59.944755 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-229.ec2.internal\" not found" Jul 16 17:28:59.990031 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:28:59.990003 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:29:00.047669 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.047641 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" Jul 16 17:29:00.062890 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.062862 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:29:00.063671 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.063658 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" Jul 16 17:29:00.074362 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.074341 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:29:00.531706 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.531674 2582 apiserver.go:52] "Watching apiserver" Jul 16 17:29:00.541003 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.540975 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 17:29:00.541439 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.541411 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tmd98","openshift-dns/node-resolver-c2rnv","openshift-network-operator/iptables-alerter-hl8xv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx","openshift-image-registry/node-ca-88s2w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal","openshift-multus/multus-4w8ml","openshift-multus/multus-additional-cni-plugins-n6flz","openshift-multus/network-metrics-daemon-ns6rx","openshift-network-diagnostics/network-check-target-zc7w4","openshift-ovn-kubernetes/ovnkube-node-642ls","kube-system/global-pull-secret-syncer-f2hc6","kube-system/konnectivity-agent-nxml2"] Jul 16 17:29:00.544139 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.544112 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:00.544306 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.544206 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:00.546490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.546464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.546592 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.546557 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:00.548462 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.548432 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 17:29:00.548750 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.548729 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.551043 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.551018 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.551561 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.551543 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 17:29:00.551892 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.551828 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 17:29:00.551892 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.551880 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2g76n\"" Jul 16 17:29:00.553708 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.553687 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 17:29:00.553825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.553754 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:29:00.553825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.553799 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 17:29:00.553904 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.553693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.554253 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.554236 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-psw84\"" Jul 16 17:29:00.556206 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.555833 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.556410 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.556376 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 17:29:00.556651 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.556633 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 17:29:00.557763 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.557744 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 17:29:00.558318 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558304 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.558412 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558365 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xf92h\"" Jul 16 17:29:00.558731 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558711 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8dd60e07-b295-4af0-b21b-28022c903e65-dbus\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.558888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.558888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558775 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-hosts-file\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.558888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnn7\" (UniqueName: \"kubernetes.io/projected/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-kube-api-access-2mnn7\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.558888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558817 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 17:29:00.558888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-tmp-dir\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.558888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558864 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6qwfs\"" Jul 16 17:29:00.558888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558877 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:00.559256 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558891 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 17:29:00.559256 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpj6\" (UniqueName: \"kubernetes.io/projected/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-kube-api-access-6cpj6\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:00.559256 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558906 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:29:00.559256 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.558925 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8dd60e07-b295-4af0-b21b-28022c903e65-kubelet-config\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.560940 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.560916 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 17:29:00.561048 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.560978 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 17:29:00.561110 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.561076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:00.561441 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.561416 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 17:29:00.561732 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.561713 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:29:00.561807 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.561785 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 17:29:00.562163 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.562134 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 17:29:00.562402 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.562385 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hgbkp\"" Jul 16 17:29:00.563677 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.563637 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.563967 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.563799 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 17:29:00.563967 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.563842 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zrrdn\"" Jul 16 17:29:00.564162 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.564022 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 17:29:00.566220 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.565970 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.566727 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.566626 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 17:29:00.566727 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.566716 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:29:00.566879 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.566862 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2khlh\"" Jul 16 17:29:00.568398 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.568379 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:00.568493 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.568454 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:00.569245 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.569210 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 17:29:00.569361 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.569319 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6rwrk\"" Jul 16 17:29:00.573871 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.571182 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.575165 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.575143 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 17:29:00.575165 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.575161 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 17:29:00.575406 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.575148 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 17:29:00.576925 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.576674 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 17:29:00.576925 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.576724 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 17:29:00.576925 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.576779 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b527r\"" Jul 16 17:29:00.576925 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.576911 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 17:29:00.601071 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.601047 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5fvmr" Jul 16 17:29:00.611027 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.610983 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5fvmr" Jul 16 17:29:00.648801 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.648774 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 17:29:00.649655 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.649601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" event={"ID":"d8456db5991817cae9ae3a70301fb5f2","Type":"ContainerStarted","Data":"b211bdc927eab2c3991ed483e3801789d35b507375f0f3cdcf862da9abcd5da9"} Jul 16 17:29:00.659943 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.659898 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.659943 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.659950 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03982997-2f7b-4052-9412-e2a38a62d542-host-slash\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.660168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.659975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysconfig\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.660168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-tmp-dir\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.660168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660036 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-cni-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.660168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660074 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppd7x\" (UniqueName: \"kubernetes.io/projected/203eb05a-d2ed-4339-b500-27085229ff5a-kube-api-access-ppd7x\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.660168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljpj\" (UniqueName: \"kubernetes.io/projected/03982997-2f7b-4052-9412-e2a38a62d542-kube-api-access-mljpj\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.660168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660124 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.660168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.660168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-conf-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-node-log\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-cni-netd\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660267 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtkl6\" (UniqueName: \"kubernetes.io/projected/fb1d58b2-bf6c-4716-930b-87379b5ded60-kube-api-access-rtkl6\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660370 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-tmp-dir\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660426 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-modprobe-d\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.660447 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-device-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.660525 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs podName:4d698729-878f-4bb3-a8f6-afd4f2a95ba9 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:01.16049145 +0000 UTC m=+2.912130993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs") pod "network-metrics-daemon-ns6rx" (UID: "4d698729-878f-4bb3-a8f6-afd4f2a95ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660568 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-k8s-cni-cncf-io\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.660604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-run\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660624 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-lib-modules\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-etc-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8dd60e07-b295-4af0-b21b-28022c903e65-kubelet-config\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660707 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggrf\" (UniqueName: \"kubernetes.io/projected/92d28cbb-bc47-4f30-a15a-511672d4b753-kube-api-access-7ggrf\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-ovnkube-config\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660742 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8dd60e07-b295-4af0-b21b-28022c903e65-kubelet-config\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660789 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysctl-d\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-var-lib-kubelet\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-host\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92d28cbb-bc47-4f30-a15a-511672d4b753-tmp\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6703413-e169-4ae5-a4a6-c6b6d93df642-agent-certs\") pod \"konnectivity-agent-nxml2\" (UID: \"c6703413-e169-4ae5-a4a6-c6b6d93df642\") " pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6703413-e169-4ae5-a4a6-c6b6d93df642-konnectivity-ca\") pod \"konnectivity-agent-nxml2\" (UID: \"c6703413-e169-4ae5-a4a6-c6b6d93df642\") " pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660953 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-sys\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660973 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-sys-fs\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.661116 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.660995 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-kubelet\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-systemd-units\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661033 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-cni-bin\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661058 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-os-release\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-hosts-file\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-socket-dir-parent\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661135 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-daemon-config\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661167 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-hosts-file\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-run-netns\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8dd60e07-b295-4af0-b21b-28022c903e65-dbus\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-cni-binary-copy\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-cni-bin\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d9c5\" (UniqueName: \"kubernetes.io/projected/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-kube-api-access-7d9c5\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8dd60e07-b295-4af0-b21b-28022c903e65-dbus\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-system-cni-dir\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.661769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnn7\" (UniqueName: \"kubernetes.io/projected/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-kube-api-access-2mnn7\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-kubernetes\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661456 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-socket-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-slash\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.661490 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-ovnkube-script-lib\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/636d2c81-02e6-4235-a858-d3993e5dba2c-host\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.661552 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret podName:8dd60e07-b295-4af0-b21b-28022c903e65 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:01.161533181 +0000 UTC m=+2.913172721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret") pod "global-pull-secret-syncer-f2hc6" (UID: "8dd60e07-b295-4af0-b21b-28022c903e65") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-cnibin\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-tuned\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-registration-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-cni-multus\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-kubelet\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661693 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-etc-kubernetes\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661717 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-ovn\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-log-socket\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.662469 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysctl-conf\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-var-lib-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-env-overrides\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661891 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/636d2c81-02e6-4235-a858-d3993e5dba2c-serviceca\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-systemd\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661940 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crq8t\" (UniqueName: \"kubernetes.io/projected/10a283b7-04b9-4a3e-871d-2ee479a07643-kube-api-access-crq8t\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.661971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-system-cni-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-cnibin\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-os-release\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-netns\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662127 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-systemd\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662156 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-multus-certs\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/203eb05a-d2ed-4339-b500-27085229ff5a-ovn-node-metrics-cert\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpj6\" (UniqueName: \"kubernetes.io/projected/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-kube-api-access-6cpj6\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:00.663038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-hostroot\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.663556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662250 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03982997-2f7b-4052-9412-e2a38a62d542-iptables-alerter-script\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.663556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkpj\" (UniqueName: \"kubernetes.io/projected/636d2c81-02e6-4235-a858-d3993e5dba2c-kube-api-access-tfkpj\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.663556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.662322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.672950 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.672905 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 17:29:00.676749 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.676720 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpj6\" (UniqueName: \"kubernetes.io/projected/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-kube-api-access-6cpj6\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:00.676749 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.676738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnn7\" (UniqueName: \"kubernetes.io/projected/42b15b11-e94a-4155-b9b0-1e5f5cf58c92-kube-api-access-2mnn7\") pod \"node-resolver-c2rnv\" (UID: \"42b15b11-e94a-4155-b9b0-1e5f5cf58c92\") " pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.762626 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-cnibin\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.762819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-tuned\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.762819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-registration-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.762819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-cni-multus\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.762819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-cnibin\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.762819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762740 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-kubelet\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.762819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-kubelet\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.762819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762773 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-registration-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.762819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762798 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-cni-multus\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762823 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-etc-kubernetes\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-ovn\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-log-socket\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762915 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-ovn\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762918 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-etc-kubernetes\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysctl-conf\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-var-lib-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-log-socket\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.762983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-env-overrides\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763028 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/636d2c81-02e6-4235-a858-d3993e5dba2c-serviceca\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763035 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-systemd\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-var-lib-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crq8t\" (UniqueName: \"kubernetes.io/projected/10a283b7-04b9-4a3e-871d-2ee479a07643-kube-api-access-crq8t\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysctl-conf\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-system-cni-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.763192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-cnibin\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763162 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-system-cni-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-os-release\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-netns\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-cnibin\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763119 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-systemd\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763217 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-systemd\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763259 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-os-release\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-multus-certs\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-netns\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/203eb05a-d2ed-4339-b500-27085229ff5a-ovn-node-metrics-cert\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763349 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-systemd\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763354 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-hostroot\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03982997-2f7b-4052-9412-e2a38a62d542-iptables-alerter-script\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763403 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkpj\" (UniqueName: \"kubernetes.io/projected/636d2c81-02e6-4235-a858-d3993e5dba2c-kube-api-access-tfkpj\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763426 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763448 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.764037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03982997-2f7b-4052-9412-e2a38a62d542-host-slash\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysconfig\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763507 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-cni-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppd7x\" (UniqueName: \"kubernetes.io/projected/203eb05a-d2ed-4339-b500-27085229ff5a-kube-api-access-ppd7x\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-multus-certs\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljpj\" (UniqueName: \"kubernetes.io/projected/03982997-2f7b-4052-9412-e2a38a62d542-kube-api-access-mljpj\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/636d2c81-02e6-4235-a858-d3993e5dba2c-serviceca\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763576 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763603 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-hostroot\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-conf-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-node-log\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763671 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysconfig\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-cni-netd\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-env-overrides\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.765045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763751 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03982997-2f7b-4052-9412-e2a38a62d542-host-slash\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtkl6\" (UniqueName: \"kubernetes.io/projected/fb1d58b2-bf6c-4716-930b-87379b5ded60-kube-api-access-rtkl6\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763808 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-modprobe-d\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763824 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-conf-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-device-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763874 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-cni-dir\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-k8s-cni-cncf-io\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763892 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-node-log\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-run\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-lib-modules\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763936 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-cni-netd\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763946 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-modprobe-d\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763989 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-device-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.763998 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764022 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-lib-modules\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-run\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764043 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-run-k8s-cni-cncf-io\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.765913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-etc-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764081 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-etc-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggrf\" (UniqueName: \"kubernetes.io/projected/92d28cbb-bc47-4f30-a15a-511672d4b753-kube-api-access-7ggrf\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-ovnkube-config\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysctl-d\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-var-lib-kubelet\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-var-lib-kubelet\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-host\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764327 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-sysctl-d\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92d28cbb-bc47-4f30-a15a-511672d4b753-tmp\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03982997-2f7b-4052-9412-e2a38a62d542-iptables-alerter-script\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6703413-e169-4ae5-a4a6-c6b6d93df642-agent-certs\") pod \"konnectivity-agent-nxml2\" (UID: \"c6703413-e169-4ae5-a4a6-c6b6d93df642\") " pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764403 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6703413-e169-4ae5-a4a6-c6b6d93df642-konnectivity-ca\") pod \"konnectivity-agent-nxml2\" (UID: \"c6703413-e169-4ae5-a4a6-c6b6d93df642\") " pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764378 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-host\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764431 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-sys\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.767105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-sys-fs\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-kubelet\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-systemd-units\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-cni-bin\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-os-release\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-socket-dir-parent\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-daemon-config\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-run-netns\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764667 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-ovnkube-config\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-cni-binary-copy\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-cni-bin\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764738 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d9c5\" (UniqueName: \"kubernetes.io/projected/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-kube-api-access-7d9c5\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-system-cni-dir\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764841 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-kubernetes\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-socket-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764893 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-slash\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.768399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-ovnkube-script-lib\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764939 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/636d2c81-02e6-4235-a858-d3993e5dba2c-host\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.764995 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765008 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765018 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/636d2c81-02e6-4235-a858-d3993e5dba2c-host\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765017 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb1d58b2-bf6c-4716-930b-87379b5ded60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765053 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-host-var-lib-cni-bin\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765103 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6703413-e169-4ae5-a4a6-c6b6d93df642-konnectivity-ca\") pod \"konnectivity-agent-nxml2\" (UID: \"c6703413-e169-4ae5-a4a6-c6b6d93df642\") " pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-socket-dir\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765157 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-run-openvswitch\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765203 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-cni-binary-copy\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765157 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-systemd-units\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-slash\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-kubelet\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-system-cni-dir\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765208 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-kubernetes\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.769171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765315 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d28cbb-bc47-4f30-a15a-511672d4b753-sys\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765308 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb1d58b2-bf6c-4716-930b-87379b5ded60-os-release\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10a283b7-04b9-4a3e-871d-2ee479a07643-sys-fs\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765361 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-cni-bin\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765363 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/203eb05a-d2ed-4339-b500-27085229ff5a-host-run-netns\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765400 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-socket-dir-parent\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765537 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/203eb05a-d2ed-4339-b500-27085229ff5a-ovnkube-script-lib\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.765650 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-multus-daemon-config\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.767243 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6703413-e169-4ae5-a4a6-c6b6d93df642-agent-certs\") pod \"konnectivity-agent-nxml2\" (UID: \"c6703413-e169-4ae5-a4a6-c6b6d93df642\") " pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.767502 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92d28cbb-bc47-4f30-a15a-511672d4b753-etc-tuned\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.767574 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/203eb05a-d2ed-4339-b500-27085229ff5a-ovn-node-metrics-cert\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.769975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.767635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92d28cbb-bc47-4f30-a15a-511672d4b753-tmp\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.772748 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.772707 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:29:00.772748 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.772740 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:29:00.772748 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.772754 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6j252 for pod openshift-network-diagnostics/network-check-target-zc7w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:00.772944 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:00.772834 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252 podName:8e8d5211-4047-48d3-89f6-bb0f78f1775f nodeName:}" failed. No retries permitted until 2026-07-16 17:29:01.272813872 +0000 UTC m=+3.024453432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6j252" (UniqueName: "kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252") pod "network-check-target-zc7w4" (UID: "8e8d5211-4047-48d3-89f6-bb0f78f1775f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:00.776452 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.776413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkpj\" (UniqueName: \"kubernetes.io/projected/636d2c81-02e6-4235-a858-d3993e5dba2c-kube-api-access-tfkpj\") pod \"node-ca-88s2w\" (UID: \"636d2c81-02e6-4235-a858-d3993e5dba2c\") " pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.776907 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.776877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crq8t\" (UniqueName: \"kubernetes.io/projected/10a283b7-04b9-4a3e-871d-2ee479a07643-kube-api-access-crq8t\") pod \"aws-ebs-csi-driver-node-6bhqx\" (UID: \"10a283b7-04b9-4a3e-871d-2ee479a07643\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.777860 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.777840 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:29:00.777970 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.777868 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppd7x\" (UniqueName: \"kubernetes.io/projected/203eb05a-d2ed-4339-b500-27085229ff5a-kube-api-access-ppd7x\") pod \"ovnkube-node-642ls\" (UID: \"203eb05a-d2ed-4339-b500-27085229ff5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.778562 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.778536 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtkl6\" (UniqueName: \"kubernetes.io/projected/fb1d58b2-bf6c-4716-930b-87379b5ded60-kube-api-access-rtkl6\") pod \"multus-additional-cni-plugins-n6flz\" (UID: \"fb1d58b2-bf6c-4716-930b-87379b5ded60\") " pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.778702 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.778685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljpj\" (UniqueName: \"kubernetes.io/projected/03982997-2f7b-4052-9412-e2a38a62d542-kube-api-access-mljpj\") pod \"iptables-alerter-hl8xv\" (UID: \"03982997-2f7b-4052-9412-e2a38a62d542\") " pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.779161 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.779141 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggrf\" (UniqueName: \"kubernetes.io/projected/92d28cbb-bc47-4f30-a15a-511672d4b753-kube-api-access-7ggrf\") pod \"tuned-tmd98\" (UID: \"92d28cbb-bc47-4f30-a15a-511672d4b753\") " pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.780004 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.779980 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d9c5\" (UniqueName: \"kubernetes.io/projected/df754ca1-c10d-4af7-a42e-cb9f6c179b1a-kube-api-access-7d9c5\") pod \"multus-4w8ml\" (UID: \"df754ca1-c10d-4af7-a42e-cb9f6c179b1a\") " pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.860215 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.860143 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c2rnv" Jul 16 17:29:00.871820 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.871798 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hl8xv" Jul 16 17:29:00.880642 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.880618 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" Jul 16 17:29:00.882051 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:00.882024 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03982997_2f7b_4052_9412_e2a38a62d542.slice/crio-2f26a287d2cb14bc62c49300b4729eca4153d3376487c98b0dad3e8524f6d6be WatchSource:0}: Error finding container 2f26a287d2cb14bc62c49300b4729eca4153d3376487c98b0dad3e8524f6d6be: Status 404 returned error can't find the container with id 2f26a287d2cb14bc62c49300b4729eca4153d3376487c98b0dad3e8524f6d6be Jul 16 17:29:00.883010 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:00.882990 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b15b11_e94a_4155_b9b0_1e5f5cf58c92.slice/crio-827339e90c0505ea9feb0d3063ea30e3c1f3d4543cec89ab2dc97e331c1ed97b WatchSource:0}: Error finding container 827339e90c0505ea9feb0d3063ea30e3c1f3d4543cec89ab2dc97e331c1ed97b: Status 404 returned error can't find the container with id 827339e90c0505ea9feb0d3063ea30e3c1f3d4543cec89ab2dc97e331c1ed97b Jul 16 17:29:00.886580 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.886553 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-88s2w" Jul 16 17:29:00.891602 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:00.891582 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a283b7_04b9_4a3e_871d_2ee479a07643.slice/crio-2f5e5245e7e6d365a48da64e997bf80ece7f1c02bd4c6a128b7cabd4ff2dad48 WatchSource:0}: Error finding container 2f5e5245e7e6d365a48da64e997bf80ece7f1c02bd4c6a128b7cabd4ff2dad48: Status 404 returned error can't find the container with id 2f5e5245e7e6d365a48da64e997bf80ece7f1c02bd4c6a128b7cabd4ff2dad48 Jul 16 17:29:00.893825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.893745 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6flz" Jul 16 17:29:00.896053 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:00.896026 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636d2c81_02e6_4235_a858_d3993e5dba2c.slice/crio-556ad8e84ca5780cf869dea89ea78f60b7db7f26dc87478ee5cc6d9552d3c461 WatchSource:0}: Error finding container 556ad8e84ca5780cf869dea89ea78f60b7db7f26dc87478ee5cc6d9552d3c461: Status 404 returned error can't find the container with id 556ad8e84ca5780cf869dea89ea78f60b7db7f26dc87478ee5cc6d9552d3c461 Jul 16 17:29:00.903388 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.903176 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:00.903483 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:00.903456 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1d58b2_bf6c_4716_930b_87379b5ded60.slice/crio-5922c97e61d2d14711d49d470fa5cd9ef7496c45c13ae406d558d1d51558849c WatchSource:0}: Error finding container 5922c97e61d2d14711d49d470fa5cd9ef7496c45c13ae406d558d1d51558849c: Status 404 returned error can't find the container with id 5922c97e61d2d14711d49d470fa5cd9ef7496c45c13ae406d558d1d51558849c Jul 16 17:29:00.911131 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.911108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tmd98" Jul 16 17:29:00.916751 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.916733 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4w8ml" Jul 16 17:29:00.922490 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:00.922458 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d28cbb_bc47_4f30_a15a_511672d4b753.slice/crio-5fe63b55a890ce25e316dc168936a10cbc41926d240b08b5d1571adabadadc8d WatchSource:0}: Error finding container 5fe63b55a890ce25e316dc168936a10cbc41926d240b08b5d1571adabadadc8d: Status 404 returned error can't find the container with id 5fe63b55a890ce25e316dc168936a10cbc41926d240b08b5d1571adabadadc8d Jul 16 17:29:00.925297 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:00.925252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:00.936130 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:00.936098 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203eb05a_d2ed_4339_b500_27085229ff5a.slice/crio-07ab811fc15a082282db2ccc451e818f784dbc54add176d48a4a6a52669284a8 WatchSource:0}: Error finding container 07ab811fc15a082282db2ccc451e818f784dbc54add176d48a4a6a52669284a8: Status 404 returned error can't find the container with id 07ab811fc15a082282db2ccc451e818f784dbc54add176d48a4a6a52669284a8 Jul 16 17:29:01.168008 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.167980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:01.168135 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.168031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:01.168135 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:01.168116 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:01.168135 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:01.168117 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:01.168242 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:01.168165 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs podName:4d698729-878f-4bb3-a8f6-afd4f2a95ba9 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:02.168152017 +0000 UTC m=+3.919791552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs") pod "network-metrics-daemon-ns6rx" (UID: "4d698729-878f-4bb3-a8f6-afd4f2a95ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:01.168242 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:01.168177 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret podName:8dd60e07-b295-4af0-b21b-28022c903e65 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:02.168171007 +0000 UTC m=+3.919810543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret") pod "global-pull-secret-syncer-f2hc6" (UID: "8dd60e07-b295-4af0-b21b-28022c903e65") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:01.182090 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:01.182059 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b041e55b23cc7cb3930d7a715f578cc.slice/crio-aefb37d555be7a2be77e575cb747b35124485792ed9d9619e5af2ab79219744b WatchSource:0}: Error finding container aefb37d555be7a2be77e575cb747b35124485792ed9d9619e5af2ab79219744b: Status 404 returned error can't find the container with id aefb37d555be7a2be77e575cb747b35124485792ed9d9619e5af2ab79219744b Jul 16 17:29:01.370042 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.369937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:01.370221 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:01.370119 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:29:01.370221 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:01.370138 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:29:01.370221 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:01.370151 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6j252 for pod openshift-network-diagnostics/network-check-target-zc7w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:01.370221 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:01.370211 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252 podName:8e8d5211-4047-48d3-89f6-bb0f78f1775f nodeName:}" failed. No retries permitted until 2026-07-16 17:29:02.370192472 +0000 UTC m=+4.121832024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6j252" (UniqueName: "kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252") pod "network-check-target-zc7w4" (UID: "8e8d5211-4047-48d3-89f6-bb0f78f1775f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:01.614142 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.614071 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:24:00 +0000 UTC" deadline="2027-12-28 20:27:33.096840175 +0000 UTC" Jul 16 17:29:01.614142 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.614138 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12722h58m31.482705345s" Jul 16 17:29:01.657161 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.656137 2582 generic.go:356] "Generic (PLEG): container finished" podID="d8456db5991817cae9ae3a70301fb5f2" containerID="44c9e7d700c27ee01bb2511f11252fe1abc3b5f2cd0b8d16c7229b8841e6e713" exitCode=0 Jul 16 17:29:01.657161 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.656298 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" event={"ID":"d8456db5991817cae9ae3a70301fb5f2","Type":"ContainerDied","Data":"44c9e7d700c27ee01bb2511f11252fe1abc3b5f2cd0b8d16c7229b8841e6e713"} Jul 16 17:29:01.666114 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.665239 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"07ab811fc15a082282db2ccc451e818f784dbc54add176d48a4a6a52669284a8"} Jul 16 17:29:01.669559 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.669475 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tmd98" event={"ID":"92d28cbb-bc47-4f30-a15a-511672d4b753","Type":"ContainerStarted","Data":"5fe63b55a890ce25e316dc168936a10cbc41926d240b08b5d1571adabadadc8d"} Jul 16 17:29:01.673366 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.673267 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6flz" event={"ID":"fb1d58b2-bf6c-4716-930b-87379b5ded60","Type":"ContainerStarted","Data":"5922c97e61d2d14711d49d470fa5cd9ef7496c45c13ae406d558d1d51558849c"} Jul 16 17:29:01.675243 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.675175 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-88s2w" event={"ID":"636d2c81-02e6-4235-a858-d3993e5dba2c","Type":"ContainerStarted","Data":"556ad8e84ca5780cf869dea89ea78f60b7db7f26dc87478ee5cc6d9552d3c461"} Jul 16 17:29:01.678651 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.678565 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" event={"ID":"0b041e55b23cc7cb3930d7a715f578cc","Type":"ContainerStarted","Data":"aefb37d555be7a2be77e575cb747b35124485792ed9d9619e5af2ab79219744b"} Jul 16 17:29:01.689250 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.689113 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4w8ml" event={"ID":"df754ca1-c10d-4af7-a42e-cb9f6c179b1a","Type":"ContainerStarted","Data":"ce8c1f0497119dfb782e366cf2672adaa78396331b3f1f39e9957d825b0ff406"} Jul 16 17:29:01.695362 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.695243 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nxml2" event={"ID":"c6703413-e169-4ae5-a4a6-c6b6d93df642","Type":"ContainerStarted","Data":"b299633f9f55e86344458cc3bdf874e804206ee8a0657adfc6875f8c3f50c7b1"} Jul 16 17:29:01.697794 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.697713 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" event={"ID":"10a283b7-04b9-4a3e-871d-2ee479a07643","Type":"ContainerStarted","Data":"2f5e5245e7e6d365a48da64e997bf80ece7f1c02bd4c6a128b7cabd4ff2dad48"} Jul 16 17:29:01.699399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.699313 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c2rnv" event={"ID":"42b15b11-e94a-4155-b9b0-1e5f5cf58c92","Type":"ContainerStarted","Data":"827339e90c0505ea9feb0d3063ea30e3c1f3d4543cec89ab2dc97e331c1ed97b"} Jul 16 17:29:01.703924 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:01.703809 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hl8xv" event={"ID":"03982997-2f7b-4052-9412-e2a38a62d542","Type":"ContainerStarted","Data":"2f26a287d2cb14bc62c49300b4729eca4153d3376487c98b0dad3e8524f6d6be"} Jul 16 17:29:02.176226 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.176105 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:02.176226 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.176174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:02.176483 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.176346 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:02.176483 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.176408 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret podName:8dd60e07-b295-4af0-b21b-28022c903e65 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:04.176391218 +0000 UTC m=+5.928030770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret") pod "global-pull-secret-syncer-f2hc6" (UID: "8dd60e07-b295-4af0-b21b-28022c903e65") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:02.177122 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.176922 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:02.177122 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.176981 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs podName:4d698729-878f-4bb3-a8f6-afd4f2a95ba9 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:04.176964637 +0000 UTC m=+5.928604177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs") pod "network-metrics-daemon-ns6rx" (UID: "4d698729-878f-4bb3-a8f6-afd4f2a95ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:02.377653 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.377611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:02.377862 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.377842 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:29:02.377931 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.377867 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:29:02.377931 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.377879 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6j252 for pod openshift-network-diagnostics/network-check-target-zc7w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:02.378031 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.377935 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252 podName:8e8d5211-4047-48d3-89f6-bb0f78f1775f nodeName:}" failed. No retries permitted until 2026-07-16 17:29:04.377918851 +0000 UTC m=+6.129558391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6j252" (UniqueName: "kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252") pod "network-check-target-zc7w4" (UID: "8e8d5211-4047-48d3-89f6-bb0f78f1775f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:02.614425 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.614264 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:24:00 +0000 UTC" deadline="2027-12-13 18:40:59.234728543 +0000 UTC" Jul 16 17:29:02.614425 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.614318 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12361h11m56.620414753s" Jul 16 17:29:02.648625 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.647789 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:02.648625 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.647935 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:02.648625 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.648016 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:02.648625 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.648102 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:02.648625 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.648159 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:02.648625 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:02.648221 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:02.731320 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:02.730531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" event={"ID":"d8456db5991817cae9ae3a70301fb5f2","Type":"ContainerStarted","Data":"d47aa664b7703a5f2bf4207d87ab562b5e9e34e3b7901d6be6ed0a8112ec2351"} Jul 16 17:29:04.195232 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:04.194473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:04.195232 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:04.194534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:04.195232 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.194694 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:04.195232 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.194756 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret podName:8dd60e07-b295-4af0-b21b-28022c903e65 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:08.194736697 +0000 UTC m=+9.946376237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret") pod "global-pull-secret-syncer-f2hc6" (UID: "8dd60e07-b295-4af0-b21b-28022c903e65") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:04.195232 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.195153 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:04.195232 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.195204 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs podName:4d698729-878f-4bb3-a8f6-afd4f2a95ba9 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:08.195189081 +0000 UTC m=+9.946828624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs") pod "network-metrics-daemon-ns6rx" (UID: "4d698729-878f-4bb3-a8f6-afd4f2a95ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:04.396669 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:04.396629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:04.396849 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.396775 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:29:04.396849 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.396795 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:29:04.396849 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.396809 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6j252 for pod openshift-network-diagnostics/network-check-target-zc7w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:04.397006 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.396865 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252 podName:8e8d5211-4047-48d3-89f6-bb0f78f1775f nodeName:}" failed. No retries permitted until 2026-07-16 17:29:08.396851735 +0000 UTC m=+10.148491272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6j252" (UniqueName: "kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252") pod "network-check-target-zc7w4" (UID: "8e8d5211-4047-48d3-89f6-bb0f78f1775f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:04.646634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:04.646538 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:04.646822 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.646691 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:04.648064 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:04.647754 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:04.648064 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.647910 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:04.648064 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:04.647977 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:04.652943 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:04.649964 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:06.645892 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:06.645862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:06.646173 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:06.646012 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:06.646385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:06.646366 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:06.646481 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:06.646464 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:06.646534 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:06.646499 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:06.646586 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:06.646544 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:08.233366 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:08.233307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:08.233918 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:08.233399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:08.233918 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.233483 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:08.233918 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.233513 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:08.233918 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.233558 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs podName:4d698729-878f-4bb3-a8f6-afd4f2a95ba9 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:16.233537507 +0000 UTC m=+17.985177063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs") pod "network-metrics-daemon-ns6rx" (UID: "4d698729-878f-4bb3-a8f6-afd4f2a95ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:08.233918 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.233577 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret podName:8dd60e07-b295-4af0-b21b-28022c903e65 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:16.233568163 +0000 UTC m=+17.985207704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret") pod "global-pull-secret-syncer-f2hc6" (UID: "8dd60e07-b295-4af0-b21b-28022c903e65") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:08.434723 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:08.434660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:08.434913 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.434850 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:29:08.434913 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.434880 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:29:08.434913 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.434896 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6j252 for pod openshift-network-diagnostics/network-check-target-zc7w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:08.435011 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.434970 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252 podName:8e8d5211-4047-48d3-89f6-bb0f78f1775f nodeName:}" failed. No retries permitted until 2026-07-16 17:29:16.434951714 +0000 UTC m=+18.186591263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6j252" (UniqueName: "kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252") pod "network-check-target-zc7w4" (UID: "8e8d5211-4047-48d3-89f6-bb0f78f1775f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:08.646330 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:08.646192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:08.646480 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:08.646360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:08.646480 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.646371 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:08.646480 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.646443 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:08.646480 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:08.646450 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:08.646702 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:08.646528 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:10.645677 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:10.645648 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:10.646152 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:10.645771 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:10.646152 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:10.645650 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:10.646152 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:10.645869 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:10.646152 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:10.645650 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:10.646152 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:10.645954 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:12.645985 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:12.645945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:12.645985 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:12.645972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:12.646490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:12.646035 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:12.646490 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:12.646144 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:12.646490 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:12.646219 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:12.646490 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:12.646319 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:14.646168 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:14.646133 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:14.646645 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:14.646180 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:14.646645 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:14.646133 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:14.646645 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:14.646297 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:14.646645 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:14.646409 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:14.646645 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:14.646490 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:16.297830 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:16.297790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:16.298446 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:16.297843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:16.298446 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.297956 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:16.298446 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.297956 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:16.298446 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.298019 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret podName:8dd60e07-b295-4af0-b21b-28022c903e65 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:32.298000045 +0000 UTC m=+34.049639600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret") pod "global-pull-secret-syncer-f2hc6" (UID: "8dd60e07-b295-4af0-b21b-28022c903e65") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:16.298446 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.298039 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs podName:4d698729-878f-4bb3-a8f6-afd4f2a95ba9 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:32.298028491 +0000 UTC m=+34.049668031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs") pod "network-metrics-daemon-ns6rx" (UID: "4d698729-878f-4bb3-a8f6-afd4f2a95ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:16.499043 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:16.498997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:16.499224 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.499133 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:29:16.499224 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.499161 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:29:16.499224 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.499176 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6j252 for pod openshift-network-diagnostics/network-check-target-zc7w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:16.499367 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.499236 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252 podName:8e8d5211-4047-48d3-89f6-bb0f78f1775f nodeName:}" failed. No retries permitted until 2026-07-16 17:29:32.499221957 +0000 UTC m=+34.250861496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6j252" (UniqueName: "kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252") pod "network-check-target-zc7w4" (UID: "8e8d5211-4047-48d3-89f6-bb0f78f1775f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:16.645961 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:16.645880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:16.645961 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:16.645883 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:16.646171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:16.645883 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:16.646171 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.646026 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:16.646249 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.646156 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:16.646249 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:16.646243 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:18.646776 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:18.646738 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:18.647239 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:18.646842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:18.647239 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:18.646851 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:18.647239 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:18.646979 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:18.647239 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:18.647014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:18.647239 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:18.647088 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:20.646191 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:20.646012 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:20.646191 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:20.646046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:20.646191 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:20.646134 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:20.646191 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:20.646142 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:20.646824 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:20.646225 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:20.646824 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:20.646313 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:21.771732 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.771384 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"3f0b37d534aa07b59b96dd9ebae357876e8f82db1300316ffbebad3c99a94fe9"} Jul 16 17:29:21.772431 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.771806 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"c309e8fd341065b814b72c3e9568b3667caf8b5f958423e32013b111bb48366f"} Jul 16 17:29:21.772431 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.771848 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"f1c68cef08c526323add69f4c1b6013b6bf96087f2e6d64ba5ac53f9304c3a6f"} Jul 16 17:29:21.772614 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.772594 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tmd98" event={"ID":"92d28cbb-bc47-4f30-a15a-511672d4b753","Type":"ContainerStarted","Data":"9349af8fe78bff30cb48bd68fa7c791967a0b142eaf34f5b59236dee39301734"} Jul 16 17:29:21.773810 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.773789 2582 generic.go:356] "Generic (PLEG): container finished" podID="fb1d58b2-bf6c-4716-930b-87379b5ded60" containerID="4fc512b76758745c59d1f4e1fe17420318bf51beefcfda8ce95a6688888c12a7" exitCode=0 Jul 16 17:29:21.773899 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.773853 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6flz" event={"ID":"fb1d58b2-bf6c-4716-930b-87379b5ded60","Type":"ContainerDied","Data":"4fc512b76758745c59d1f4e1fe17420318bf51beefcfda8ce95a6688888c12a7"} Jul 16 17:29:21.775327 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.775303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-88s2w" event={"ID":"636d2c81-02e6-4235-a858-d3993e5dba2c","Type":"ContainerStarted","Data":"dab23ce2ae38c7ce14212b693dce51b1ca2035bf81d8a848190ccf7248a13b37"} Jul 16 17:29:21.776711 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.776670 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" event={"ID":"0b041e55b23cc7cb3930d7a715f578cc","Type":"ContainerStarted","Data":"c1defe5e0359f80657411cd3bb9cabbc8154318af89cc423a4cec8ffd9556933"} Jul 16 17:29:21.778071 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.778050 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4w8ml" event={"ID":"df754ca1-c10d-4af7-a42e-cb9f6c179b1a","Type":"ContainerStarted","Data":"4bac505640f3998039563585e9e6eb6c3d6dae4c96afef723e074aaef2ad165a"} Jul 16 17:29:21.782859 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.782835 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nxml2" event={"ID":"c6703413-e169-4ae5-a4a6-c6b6d93df642","Type":"ContainerStarted","Data":"c7477fafd3579f4e44bf5d0ce2a24a7aef0409f52e49e54c1397bd0e0a85f4e9"} Jul 16 17:29:21.784402 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.784364 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" event={"ID":"10a283b7-04b9-4a3e-871d-2ee479a07643","Type":"ContainerStarted","Data":"1af991fc7b05e57499cd9a4d13d9c485f3beea6730c99867f76e8ff903f7982a"} Jul 16 17:29:21.785775 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.785755 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c2rnv" event={"ID":"42b15b11-e94a-4155-b9b0-1e5f5cf58c92","Type":"ContainerStarted","Data":"8e7040d154712e502470bec8c1ecea78939b1fb73f7493e6781b6a1b2382ad8b"} Jul 16 17:29:21.805746 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.805695 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-229.ec2.internal" podStartSLOduration=21.805680191 podStartE2EDuration="21.805680191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:02.756805379 +0000 UTC m=+4.508444940" watchObservedRunningTime="2026-07-16 17:29:21.805680191 +0000 UTC m=+23.557319748" Jul 16 17:29:21.855494 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.855439 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4w8ml" podStartSLOduration=3.474779233 podStartE2EDuration="23.855423268s" totalImagesPullingTime="20.380644035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:21.855056394 +0000 UTC m=+23.606695952" watchObservedRunningTime="2026-07-16 17:29:21.855423268 +0000 UTC m=+23.607062825" Jul 16 17:29:21.856079 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.856048 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-88s2w" podStartSLOduration=7.995511832 podStartE2EDuration="23.856039223s" totalImagesPullingTime="15.860527391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:21.832767875 +0000 UTC m=+23.584407454" watchObservedRunningTime="2026-07-16 17:29:21.856039223 +0000 UTC m=+23.607678781" Jul 16 17:29:21.885130 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.884982 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nxml2" podStartSLOduration=3.550438146 podStartE2EDuration="23.884966273s" totalImagesPullingTime="20.334528127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:21.884251887 +0000 UTC m=+23.635891445" watchObservedRunningTime="2026-07-16 17:29:21.884966273 +0000 UTC m=+23.636605830" Jul 16 17:29:21.906338 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.906292 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tmd98" podStartSLOduration=3.571874235 podStartE2EDuration="23.906256295s" totalImagesPullingTime="20.33438206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:21.906090592 +0000 UTC m=+23.657730150" watchObservedRunningTime="2026-07-16 17:29:21.906256295 +0000 UTC m=+23.657895852" Jul 16 17:29:21.989406 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.989375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:21.990348 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:21.990316 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-229.ec2.internal" podStartSLOduration=21.990301691 podStartE2EDuration="21.990301691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:21.933697672 +0000 UTC m=+23.685337232" watchObservedRunningTime="2026-07-16 17:29:21.990301691 +0000 UTC m=+23.741941246" Jul 16 17:29:22.150895 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.150849 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:22.252513 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.252426 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:22.360903 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.360871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:22.460737 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.460704 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:22.574298 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.574253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:22.646056 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.646024 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:22.646056 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.646051 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:22.646293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.646063 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:22.646293 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:22.646151 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:22.646293 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:22.646215 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:22.646293 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:22.646274 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:22.668919 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.668888 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:22.792783 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.792746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"0053c3912a24e08d88c833a37cc6dd2195dd176e73a3a69ae0ee402d6b6c18f1"} Jul 16 17:29:22.793180 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.792847 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"d27c4a26ad278b7fc46792d4f1660f904b73e13a3f61d5f944e508192fcb92c7"} Jul 16 17:29:22.793180 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.792902 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"51fcb61d4c2c494d2621f7b8af4661417d469f184098bacbd0f03a39c1abceb4"} Jul 16 17:29:22.794428 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.794396 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hl8xv" event={"ID":"03982997-2f7b-4052-9412-e2a38a62d542","Type":"ContainerStarted","Data":"00bc3f5023fe14e18dc69994e6be5b4f5fd6520f76f5a2954b947e80aa33e9eb"} Jul 16 17:29:22.823543 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.823491 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c2rnv" podStartSLOduration=4.4638228 podStartE2EDuration="24.823474338s" totalImagesPullingTime="20.359651538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:21.99501939 +0000 UTC m=+23.746658946" watchObservedRunningTime="2026-07-16 17:29:22.823474338 +0000 UTC m=+24.575113895" Jul 16 17:29:22.823702 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.823595 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hl8xv" podStartSLOduration=4.520811874 podStartE2EDuration="24.823588773s" totalImagesPullingTime="20.302776899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:22.823433598 +0000 UTC m=+24.575073171" watchObservedRunningTime="2026-07-16 17:29:22.823588773 +0000 UTC m=+24.575228334" Jul 16 17:29:22.890818 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.890788 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 17:29:22.952267 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:22.952233 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:23.043093 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:23.043015 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:23.374915 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:23.374832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:23.447695 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:23.447654 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:23.617463 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:23.617353 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:29:22.890812485Z","UUID":"71e309e6-1324-4138-a41b-10fd9023c90d","Handler":null,"Name":"","Endpoint":""} Jul 16 17:29:23.619033 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:23.619009 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 17:29:23.619189 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:23.619064 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 17:29:23.798743 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:23.798658 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" event={"ID":"10a283b7-04b9-4a3e-871d-2ee479a07643","Type":"ContainerStarted","Data":"7995cb8869a00dc77ba08f62158050f902706034e81183aede9dca0545403815"} Jul 16 17:29:24.412825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.412571 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:24.413241 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.413223 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:24.645492 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.645459 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:24.645768 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.645460 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:24.645768 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.645594 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:24.645768 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:24.645588 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:24.645768 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:24.645665 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:24.645768 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:24.645749 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:24.804093 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.804010 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" event={"ID":"10a283b7-04b9-4a3e-871d-2ee479a07643","Type":"ContainerStarted","Data":"620b08231b1d87c8e2d3a832e8bd92fd4849ca8fade954b6cc79330e78132707"} Jul 16 17:29:24.807767 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.807739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"e12407209164a107483640220f39587732a6eaeb94c921c04d77cee494a13ccf"} Jul 16 17:29:24.808013 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.807995 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:24.808565 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.808546 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nxml2" Jul 16 17:29:24.851604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:24.851548 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bhqx" podStartSLOduration=4.210710831 podStartE2EDuration="26.851529744s" totalImagesPullingTime="22.640818913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:24.830986851 +0000 UTC m=+26.582626408" watchObservedRunningTime="2026-07-16 17:29:24.851529744 +0000 UTC m=+26.603169337" Jul 16 17:29:25.225790 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:25.225709 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:25.315658 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:25.315629 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:26.646014 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.645767 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:26.646692 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.645766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:26.646692 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:26.646135 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:26.646692 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.645792 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:26.646692 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:26.646187 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:26.646692 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:26.646255 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:26.816187 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.816147 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" event={"ID":"203eb05a-d2ed-4339-b500-27085229ff5a","Type":"ContainerStarted","Data":"d90bd45f6d94463ee6bba8e519ae167914a48bdd49d2849acd233c839dee1ceb"} Jul 16 17:29:26.816869 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.816644 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:26.816869 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.816745 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:26.816869 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.816815 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:26.818312 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.818247 2582 generic.go:356] "Generic (PLEG): container finished" podID="fb1d58b2-bf6c-4716-930b-87379b5ded60" containerID="827182e7dc66f18be402ecef7281a93c0c6b55b9c3565ab4d47e68701933b8bb" exitCode=0 Jul 16 17:29:26.818412 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.818320 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6flz" event={"ID":"fb1d58b2-bf6c-4716-930b-87379b5ded60","Type":"ContainerDied","Data":"827182e7dc66f18be402ecef7281a93c0c6b55b9c3565ab4d47e68701933b8bb"} Jul 16 17:29:26.835041 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.835014 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:26.839732 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.839709 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:29:26.857676 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.857631 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" podStartSLOduration=8.482839143 podStartE2EDuration="28.857617269s" totalImagesPullingTime="20.374778126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:26.857212834 +0000 UTC m=+28.608852404" watchObservedRunningTime="2026-07-16 17:29:26.857617269 +0000 UTC m=+28.609256857" Jul 16 17:29:26.912691 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:26.912660 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:27.004390 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:27.004359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:28.648460 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.648243 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:28.648965 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.648353 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:28.648965 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:28.648551 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:28.648965 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.648392 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:28.648965 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:28.648616 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:28.648965 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:28.648733 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:28.735485 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.735444 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ns6rx"] Jul 16 17:29:28.751493 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.751456 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zc7w4"] Jul 16 17:29:28.752316 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.752269 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2hc6"] Jul 16 17:29:28.824957 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.824872 2582 generic.go:356] "Generic (PLEG): container finished" podID="fb1d58b2-bf6c-4716-930b-87379b5ded60" containerID="2f9d0da06208af0431cf1756cd5a6f5e3b250efddd3c95b9d39ba9837027fd07" exitCode=0 Jul 16 17:29:28.824957 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.824947 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:28.825179 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.824972 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6flz" event={"ID":"fb1d58b2-bf6c-4716-930b-87379b5ded60","Type":"ContainerDied","Data":"2f9d0da06208af0431cf1756cd5a6f5e3b250efddd3c95b9d39ba9837027fd07"} Jul 16 17:29:28.825179 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.825150 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:28.825309 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:28.825167 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:28.825309 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:28.825225 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:28.825415 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:28.825320 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:28.825415 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:28.825372 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:30.645956 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:30.645926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:30.646576 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:30.646031 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:30.646576 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:30.646036 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:30.646576 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:30.646155 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:30.646576 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:30.646211 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:30.646576 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:30.646341 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:30.839799 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:30.839766 2582 generic.go:356] "Generic (PLEG): container finished" podID="fb1d58b2-bf6c-4716-930b-87379b5ded60" containerID="50a25783bddabd6863cbb999836a2b5b54f2df6b9dc81b99ed8de2709b10524e" exitCode=0 Jul 16 17:29:30.839981 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:30.839815 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6flz" event={"ID":"fb1d58b2-bf6c-4716-930b-87379b5ded60","Type":"ContainerDied","Data":"50a25783bddabd6863cbb999836a2b5b54f2df6b9dc81b99ed8de2709b10524e"} Jul 16 17:29:32.314526 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:32.314482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:32.315196 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:32.314552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:32.315196 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.314683 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:32.315196 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.314739 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:32.315196 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.314753 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret podName:8dd60e07-b295-4af0-b21b-28022c903e65 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:04.314736548 +0000 UTC m=+66.066376086 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret") pod "global-pull-secret-syncer-f2hc6" (UID: "8dd60e07-b295-4af0-b21b-28022c903e65") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:29:32.315196 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.314801 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs podName:4d698729-878f-4bb3-a8f6-afd4f2a95ba9 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:04.314783789 +0000 UTC m=+66.066423330 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs") pod "network-metrics-daemon-ns6rx" (UID: "4d698729-878f-4bb3-a8f6-afd4f2a95ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:29:32.516673 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:32.516633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:32.516849 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.516812 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:29:32.516849 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.516833 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:29:32.516849 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.516842 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6j252 for pod openshift-network-diagnostics/network-check-target-zc7w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:32.517002 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.516915 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252 podName:8e8d5211-4047-48d3-89f6-bb0f78f1775f nodeName:}" failed. No retries permitted until 2026-07-16 17:30:04.516885764 +0000 UTC m=+66.268525320 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6j252" (UniqueName: "kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252") pod "network-check-target-zc7w4" (UID: "8e8d5211-4047-48d3-89f6-bb0f78f1775f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:29:32.645420 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:32.645328 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:32.645420 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:32.645362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:32.645612 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.645468 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:32.645612 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.645513 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:32.645612 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:32.645565 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:32.645755 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:32.645639 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:34.646016 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:34.645974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:34.646679 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:34.645974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:34.646679 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:34.646096 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ns6rx" podUID="4d698729-878f-4bb3-a8f6-afd4f2a95ba9" Jul 16 17:29:34.646679 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:34.646183 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2hc6" podUID="8dd60e07-b295-4af0-b21b-28022c903e65" Jul 16 17:29:34.646679 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:34.645973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:34.646679 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:34.646301 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zc7w4" podUID="8e8d5211-4047-48d3-89f6-bb0f78f1775f" Jul 16 17:29:35.123826 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.123756 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-229.ec2.internal" event="NodeReady" Jul 16 17:29:35.123983 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.123897 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 17:29:35.220106 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.220073 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ztxrd"] Jul 16 17:29:35.232118 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.232085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.236778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.236756 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-27n67\"" Jul 16 17:29:35.237820 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.237803 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 17:29:35.237981 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.237853 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 17:29:35.248601 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.248576 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h6k2d"] Jul 16 17:29:35.262013 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.261985 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ztxrd"] Jul 16 17:29:35.262163 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.262030 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6k2d"] Jul 16 17:29:35.262163 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.262110 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6k2d" Jul 16 17:29:35.265549 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.265522 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 17:29:35.265679 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.265633 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4zxvh\"" Jul 16 17:29:35.266306 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.266267 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 17:29:35.272543 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.272519 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 17:29:35.338407 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.338370 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.338549 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.338425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca0ed375-c645-4cc1-9264-67043142deb5-config-volume\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.338549 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.338476 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0ed375-c645-4cc1-9264-67043142deb5-tmp-dir\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.338632 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.338596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cac65f2-474b-4aa7-80d9-a7530cebfbf5-cert\") pod \"ingress-canary-h6k2d\" (UID: \"8cac65f2-474b-4aa7-80d9-a7530cebfbf5\") " pod="openshift-ingress-canary/ingress-canary-h6k2d" Jul 16 17:29:35.338632 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.338618 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkwx\" (UniqueName: \"kubernetes.io/projected/ca0ed375-c645-4cc1-9264-67043142deb5-kube-api-access-4wkwx\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.338708 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.338637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8pb6\" (UniqueName: \"kubernetes.io/projected/8cac65f2-474b-4aa7-80d9-a7530cebfbf5-kube-api-access-z8pb6\") pod \"ingress-canary-h6k2d\" (UID: \"8cac65f2-474b-4aa7-80d9-a7530cebfbf5\") " pod="openshift-ingress-canary/ingress-canary-h6k2d" Jul 16 17:29:35.440607 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.439412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cac65f2-474b-4aa7-80d9-a7530cebfbf5-cert\") pod \"ingress-canary-h6k2d\" (UID: \"8cac65f2-474b-4aa7-80d9-a7530cebfbf5\") " pod="openshift-ingress-canary/ingress-canary-h6k2d" Jul 16 17:29:35.440607 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.439464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkwx\" (UniqueName: \"kubernetes.io/projected/ca0ed375-c645-4cc1-9264-67043142deb5-kube-api-access-4wkwx\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.440607 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.439488 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8pb6\" (UniqueName: \"kubernetes.io/projected/8cac65f2-474b-4aa7-80d9-a7530cebfbf5-kube-api-access-z8pb6\") pod \"ingress-canary-h6k2d\" (UID: \"8cac65f2-474b-4aa7-80d9-a7530cebfbf5\") " pod="openshift-ingress-canary/ingress-canary-h6k2d" Jul 16 17:29:35.440607 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.439534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.440607 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.439586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca0ed375-c645-4cc1-9264-67043142deb5-config-volume\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.440607 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.439618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0ed375-c645-4cc1-9264-67043142deb5-tmp-dir\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.440607 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.439937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0ed375-c645-4cc1-9264-67043142deb5-tmp-dir\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.441036 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:35.440892 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:29:35.441036 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:35.440955 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls podName:ca0ed375-c645-4cc1-9264-67043142deb5 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:35.940935583 +0000 UTC m=+37.692575133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls") pod "dns-default-ztxrd" (UID: "ca0ed375-c645-4cc1-9264-67043142deb5") : secret "dns-default-metrics-tls" not found Jul 16 17:29:35.441636 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.441610 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca0ed375-c645-4cc1-9264-67043142deb5-config-volume\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.445355 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.445200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cac65f2-474b-4aa7-80d9-a7530cebfbf5-cert\") pod \"ingress-canary-h6k2d\" (UID: \"8cac65f2-474b-4aa7-80d9-a7530cebfbf5\") " pod="openshift-ingress-canary/ingress-canary-h6k2d" Jul 16 17:29:35.460975 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.460952 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8pb6\" (UniqueName: \"kubernetes.io/projected/8cac65f2-474b-4aa7-80d9-a7530cebfbf5-kube-api-access-z8pb6\") pod \"ingress-canary-h6k2d\" (UID: \"8cac65f2-474b-4aa7-80d9-a7530cebfbf5\") " pod="openshift-ingress-canary/ingress-canary-h6k2d" Jul 16 17:29:35.462460 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.462445 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkwx\" (UniqueName: \"kubernetes.io/projected/ca0ed375-c645-4cc1-9264-67043142deb5-kube-api-access-4wkwx\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.571297 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.571248 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6k2d" Jul 16 17:29:35.612811 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.612783 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:35.718409 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.718375 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6k2d"] Jul 16 17:29:35.725900 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:35.725870 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cac65f2_474b_4aa7_80d9_a7530cebfbf5.slice/crio-a21debf634cdb7577facab950923bf263ab27c63af89d27e156899e28d8a350d WatchSource:0}: Error finding container a21debf634cdb7577facab950923bf263ab27c63af89d27e156899e28d8a350d: Status 404 returned error can't find the container with id a21debf634cdb7577facab950923bf263ab27c63af89d27e156899e28d8a350d Jul 16 17:29:35.766250 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.766192 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:35.854154 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.854113 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6k2d" event={"ID":"8cac65f2-474b-4aa7-80d9-a7530cebfbf5","Type":"ContainerStarted","Data":"a21debf634cdb7577facab950923bf263ab27c63af89d27e156899e28d8a350d"} Jul 16 17:29:35.856848 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.856820 2582 generic.go:356] "Generic (PLEG): container finished" podID="fb1d58b2-bf6c-4716-930b-87379b5ded60" containerID="be6abcb91582d9044ac4f35993587474cebf59581606eac6f7487adbe599224d" exitCode=0 Jul 16 17:29:35.856985 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.856868 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6flz" event={"ID":"fb1d58b2-bf6c-4716-930b-87379b5ded60","Type":"ContainerDied","Data":"be6abcb91582d9044ac4f35993587474cebf59581606eac6f7487adbe599224d"} Jul 16 17:29:35.944547 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:35.944514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:35.944695 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:35.944669 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:29:35.944830 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:35.944736 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls podName:ca0ed375-c645-4cc1-9264-67043142deb5 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:36.944716046 +0000 UTC m=+38.696355599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls") pod "dns-default-ztxrd" (UID: "ca0ed375-c645-4cc1-9264-67043142deb5") : secret "dns-default-metrics-tls" not found Jul 16 17:29:36.645995 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.645957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:29:36.646178 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.645961 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:29:36.646178 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.646078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:29:36.650639 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.650614 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:29:36.650639 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.650623 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:29:36.650639 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.650614 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:29:36.650904 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.650614 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lggz\"" Jul 16 17:29:36.650904 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.650655 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vscvt\"" Jul 16 17:29:36.650904 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.650616 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:29:36.863842 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.863641 2582 generic.go:356] "Generic (PLEG): container finished" podID="fb1d58b2-bf6c-4716-930b-87379b5ded60" containerID="b4bc004791cf3ebf9fbf433cd706102bc5ef255f9334e72f2e83d919bb6fbe72" exitCode=0 Jul 16 17:29:36.863842 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.863742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6flz" event={"ID":"fb1d58b2-bf6c-4716-930b-87379b5ded60","Type":"ContainerDied","Data":"b4bc004791cf3ebf9fbf433cd706102bc5ef255f9334e72f2e83d919bb6fbe72"} Jul 16 17:29:36.950171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:36.950150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:36.950351 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:36.950333 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:29:36.950417 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:36.950404 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls podName:ca0ed375-c645-4cc1-9264-67043142deb5 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:38.950383104 +0000 UTC m=+40.702022654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls") pod "dns-default-ztxrd" (UID: "ca0ed375-c645-4cc1-9264-67043142deb5") : secret "dns-default-metrics-tls" not found Jul 16 17:29:37.870991 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:37.870952 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6flz" event={"ID":"fb1d58b2-bf6c-4716-930b-87379b5ded60","Type":"ContainerStarted","Data":"1967f370637d1d4e36cba7cdffce1be87c64a08ffc85162353f85417a53e3e83"} Jul 16 17:29:37.904738 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:37.901862 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n6flz" podStartSLOduration=12.510860855 podStartE2EDuration="39.901841006s" totalImagesPullingTime="27.390980151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:37.899700772 +0000 UTC m=+39.651340349" watchObservedRunningTime="2026-07-16 17:29:37.901841006 +0000 UTC m=+39.653480565" Jul 16 17:29:38.067588 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:38.067556 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:38.177092 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:38.177009 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:38.274839 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:38.274805 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:38.874916 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:38.874880 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6k2d" event={"ID":"8cac65f2-474b-4aa7-80d9-a7530cebfbf5","Type":"ContainerStarted","Data":"0b2e7125c6b52a5cde363e5901bb75742263f98ff70bf926a53f2ebabaf8dd79"} Jul 16 17:29:38.904449 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:38.904349 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h6k2d" podStartSLOduration=0.997454642 podStartE2EDuration="3.904328875s" totalImagesPullingTime="2.906874233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:38.904020478 +0000 UTC m=+40.655660038" watchObservedRunningTime="2026-07-16 17:29:38.904328875 +0000 UTC m=+40.655968468" Jul 16 17:29:38.952334 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:38.952307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:38.966408 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:38.966382 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:38.966616 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:38.966587 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:29:38.966689 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:38.966669 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls podName:ca0ed375-c645-4cc1-9264-67043142deb5 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:42.966647879 +0000 UTC m=+44.718287431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls") pod "dns-default-ztxrd" (UID: "ca0ed375-c645-4cc1-9264-67043142deb5") : secret "dns-default-metrics-tls" not found Jul 16 17:29:39.051170 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:39.051138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:40.652580 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.652546 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc"] Jul 16 17:29:40.655559 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.655535 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc" Jul 16 17:29:40.658592 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.658551 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2sj4w\"" Jul 16 17:29:40.659374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.659352 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5"] Jul 16 17:29:40.663004 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.662982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5" Jul 16 17:29:40.665757 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.665727 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 16 17:29:40.665867 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.665830 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:29:40.665867 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.665846 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kq7nq\"" Jul 16 17:29:40.668372 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.668348 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc"] Jul 16 17:29:40.673453 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.673425 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5"] Jul 16 17:29:40.764665 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.764627 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg"] Jul 16 17:29:40.767585 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.767561 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:40.770621 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.770601 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr"] Jul 16 17:29:40.770809 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.770786 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 16 17:29:40.770906 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.770835 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2j945\"" Jul 16 17:29:40.770906 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.770825 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 17:29:40.771023 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.770962 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 17:29:40.773152 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.773130 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 16 17:29:40.773378 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.773361 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.775616 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.775599 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 16 17:29:40.776189 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.776172 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 16 17:29:40.776314 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.776205 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 16 17:29:40.776314 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.776217 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zzpms\"" Jul 16 17:29:40.776314 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.776263 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:29:40.777729 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.777711 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57cf54b466-4nz2b"] Jul 16 17:29:40.779558 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.779531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8lk\" (UniqueName: \"kubernetes.io/projected/38a67d8c-f26e-4d5e-b772-2ea4db4b6904-kube-api-access-hc8lk\") pod \"volume-data-source-validator-78495597f6-gfhb5\" (UID: \"38a67d8c-f26e-4d5e-b772-2ea4db4b6904\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5" Jul 16 17:29:40.779662 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.779566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4mn\" (UniqueName: \"kubernetes.io/projected/e915e1a6-38b0-47c0-b9b0-47d4ed2c4809-kube-api-access-xc4mn\") pod \"network-check-source-869dcfcdcd-h9pdc\" (UID: \"e915e1a6-38b0-47c0-b9b0-47d4ed2c4809\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc" Jul 16 17:29:40.780566 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.780545 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c7798b5fb-gw7rk"] Jul 16 17:29:40.780690 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.780673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.783356 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.783336 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg"] Jul 16 17:29:40.783463 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.783448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.785339 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.785249 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 17:29:40.786373 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.786348 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr"] Jul 16 17:29:40.787135 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.787110 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 17:29:40.787135 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.787130 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 16 17:29:40.787309 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.787154 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 17:29:40.787551 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.787527 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 16 17:29:40.787823 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.787803 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 16 17:29:40.787915 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.787894 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 17:29:40.788313 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.788298 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 16 17:29:40.789252 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.788732 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nc8wp\"" Jul 16 17:29:40.789252 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.788923 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 17:29:40.789665 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.789645 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wm2ng\"" Jul 16 17:29:40.793833 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.793815 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 17:29:40.802016 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.801991 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57cf54b466-4nz2b"] Jul 16 17:29:40.806187 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.806161 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c7798b5fb-gw7rk"] Jul 16 17:29:40.880904 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.880876 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8lk\" (UniqueName: \"kubernetes.io/projected/38a67d8c-f26e-4d5e-b772-2ea4db4b6904-kube-api-access-hc8lk\") pod \"volume-data-source-validator-78495597f6-gfhb5\" (UID: \"38a67d8c-f26e-4d5e-b772-2ea4db4b6904\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5" Jul 16 17:29:40.881087 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.880919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-installation-pull-secrets\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.881087 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.880947 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rgv\" (UniqueName: \"kubernetes.io/projected/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-kube-api-access-n4rgv\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:40.881087 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.880985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.881200 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-image-registry-private-configuration\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.881200 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:40.881200 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-trusted-ca\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.881342 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.881342 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881218 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-stats-auth\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.881342 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881243 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-certificates\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.881342 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx48g\" (UniqueName: \"kubernetes.io/projected/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-kube-api-access-xx48g\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.881474 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4mn\" (UniqueName: \"kubernetes.io/projected/e915e1a6-38b0-47c0-b9b0-47d4ed2c4809-kube-api-access-xc4mn\") pod \"network-check-source-869dcfcdcd-h9pdc\" (UID: \"e915e1a6-38b0-47c0-b9b0-47d4ed2c4809\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc" Jul 16 17:29:40.881474 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xh4\" (UniqueName: \"kubernetes.io/projected/e4264197-b690-4ca8-a31f-c6c32cf8eba8-kube-api-access-g6xh4\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.881474 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-bound-sa-token\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.881474 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:40.881601 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-config\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.881601 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.881601 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50a4b64a-aadc-46c6-926d-5148fcd994fd-ca-trust-extracted\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.881601 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmz8\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-kube-api-access-thmz8\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.881712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-default-certificate\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.881712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.881631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.904871 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.904803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4mn\" (UniqueName: \"kubernetes.io/projected/e915e1a6-38b0-47c0-b9b0-47d4ed2c4809-kube-api-access-xc4mn\") pod \"network-check-source-869dcfcdcd-h9pdc\" (UID: \"e915e1a6-38b0-47c0-b9b0-47d4ed2c4809\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc" Jul 16 17:29:40.904990 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.904900 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8lk\" (UniqueName: \"kubernetes.io/projected/38a67d8c-f26e-4d5e-b772-2ea4db4b6904-kube-api-access-hc8lk\") pod \"volume-data-source-validator-78495597f6-gfhb5\" (UID: \"38a67d8c-f26e-4d5e-b772-2ea4db4b6904\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5" Jul 16 17:29:40.969006 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.968967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc" Jul 16 17:29:40.975215 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.975184 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5" Jul 16 17:29:40.982557 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.982530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.982648 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.982591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-image-registry-private-configuration\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.982648 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.982633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:40.982749 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.982660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-trusted-ca\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.982749 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.982685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.982749 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.982709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-stats-auth\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.982884 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:40.982786 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:40.982945 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:40.982929 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls podName:8debfe7c-6fd2-4ed1-a47b-a916cd09d836 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:41.48289458 +0000 UTC m=+43.234534136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-zvnlg" (UID: "8debfe7c-6fd2-4ed1-a47b-a916cd09d836") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:40.983109 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:40.983088 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:41.483064666 +0000 UTC m=+43.234704204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : configmap references non-existent config key: service-ca.crt Jul 16 17:29:40.983195 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983166 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-certificates\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.983248 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx48g\" (UniqueName: \"kubernetes.io/projected/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-kube-api-access-xx48g\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.983248 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xh4\" (UniqueName: \"kubernetes.io/projected/e4264197-b690-4ca8-a31f-c6c32cf8eba8-kube-api-access-g6xh4\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.983376 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-bound-sa-token\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.983376 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:40.983376 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-config\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.983529 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.983529 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50a4b64a-aadc-46c6-926d-5148fcd994fd-ca-trust-extracted\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.983529 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983442 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmz8\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-kube-api-access-thmz8\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.983529 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-default-certificate\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.983529 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.983529 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-installation-pull-secrets\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.983815 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rgv\" (UniqueName: \"kubernetes.io/projected/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-kube-api-access-n4rgv\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:40.983815 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.983754 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-certificates\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.983972 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:40.983876 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:29:40.983972 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:40.983907 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf54b466-4nz2b: secret "image-registry-tls" not found Jul 16 17:29:40.983972 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:40.983953 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls podName:50a4b64a-aadc-46c6-926d-5148fcd994fd nodeName:}" failed. No retries permitted until 2026-07-16 17:29:41.483935857 +0000 UTC m=+43.235575397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls") pod "image-registry-57cf54b466-4nz2b" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd") : secret "image-registry-tls" not found Jul 16 17:29:40.984121 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.984029 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-trusted-ca\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.984297 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.984252 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50a4b64a-aadc-46c6-926d-5148fcd994fd-ca-trust-extracted\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.984392 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:40.984378 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:29:40.984477 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:40.984425 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:41.484410548 +0000 UTC m=+43.236050088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : secret "router-metrics-certs-default" not found Jul 16 17:29:40.985144 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.985112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:40.985252 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.985179 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-config\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.985880 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.985834 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-stats-auth\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.986741 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.986716 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:40.987800 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.987755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-installation-pull-secrets\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:40.987800 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.987786 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-default-certificate\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:40.987944 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:40.987847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-image-registry-private-configuration\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:41.009229 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.009193 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx48g\" (UniqueName: \"kubernetes.io/projected/50bf71f3-1d88-4cb0-ab88-7461907c4fd0-kube-api-access-xx48g\") pod \"service-ca-operator-7695fc6c8c-qh8vr\" (UID: \"50bf71f3-1d88-4cb0-ab88-7461907c4fd0\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:41.012758 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.012714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmz8\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-kube-api-access-thmz8\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:41.014455 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.014403 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-bound-sa-token\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:41.015463 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.015439 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rgv\" (UniqueName: \"kubernetes.io/projected/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-kube-api-access-n4rgv\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:41.020143 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.020117 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xh4\" (UniqueName: \"kubernetes.io/projected/e4264197-b690-4ca8-a31f-c6c32cf8eba8-kube-api-access-g6xh4\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:41.068170 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.068134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:41.084512 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.084479 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" Jul 16 17:29:41.185640 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.185611 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:41.239344 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.239314 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr"] Jul 16 17:29:41.242648 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:41.242622 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bf71f3_1d88_4cb0_ab88_7461907c4fd0.slice/crio-b3c995bd0cc32c1e281897853207a3e6649c683fd78e3a04c5f3341533c2235e WatchSource:0}: Error finding container b3c995bd0cc32c1e281897853207a3e6649c683fd78e3a04c5f3341533c2235e: Status 404 returned error can't find the container with id b3c995bd0cc32c1e281897853207a3e6649c683fd78e3a04c5f3341533c2235e Jul 16 17:29:41.321087 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.320999 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc"] Jul 16 17:29:41.323595 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:41.323560 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode915e1a6_38b0_47c0_b9b0_47d4ed2c4809.slice/crio-ef6d47bf77a8f088d66f0129e8bfea9b6894fe26d4af1b5a96f50de06f2a990e WatchSource:0}: Error finding container ef6d47bf77a8f088d66f0129e8bfea9b6894fe26d4af1b5a96f50de06f2a990e: Status 404 returned error can't find the container with id ef6d47bf77a8f088d66f0129e8bfea9b6894fe26d4af1b5a96f50de06f2a990e Jul 16 17:29:41.323744 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.323604 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5"] Jul 16 17:29:41.326885 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:29:41.326861 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a67d8c_f26e_4d5e_b772_2ea4db4b6904.slice/crio-dfc263dff1108243b07771849398747e06e9a2c624a94b5b89fa784915dd7bbc WatchSource:0}: Error finding container dfc263dff1108243b07771849398747e06e9a2c624a94b5b89fa784915dd7bbc: Status 404 returned error can't find the container with id dfc263dff1108243b07771849398747e06e9a2c624a94b5b89fa784915dd7bbc Jul 16 17:29:41.398837 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.398786 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:41.488674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.488590 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:41.488674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.488626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:41.488889 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:41.488766 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:41.488889 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:41.488841 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls podName:8debfe7c-6fd2-4ed1-a47b-a916cd09d836 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:42.488816752 +0000 UTC m=+44.240456304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-zvnlg" (UID: "8debfe7c-6fd2-4ed1-a47b-a916cd09d836") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:41.488889 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:41.488862 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:42.488847976 +0000 UTC m=+44.240487515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : configmap references non-existent config key: service-ca.crt Jul 16 17:29:41.489049 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.488911 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:41.489049 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.488933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:41.489049 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:41.489009 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:29:41.489049 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:41.489027 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:29:41.489049 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:41.489040 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf54b466-4nz2b: secret "image-registry-tls" not found Jul 16 17:29:41.489049 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:41.489048 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:42.489032911 +0000 UTC m=+44.240672447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : secret "router-metrics-certs-default" not found Jul 16 17:29:41.489272 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:41.489071 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls podName:50a4b64a-aadc-46c6-926d-5148fcd994fd nodeName:}" failed. No retries permitted until 2026-07-16 17:29:42.489062249 +0000 UTC m=+44.240701785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls") pod "image-registry-57cf54b466-4nz2b" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd") : secret "image-registry-tls" not found Jul 16 17:29:41.511991 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.511962 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:41.597327 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.597252 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" limit=1024 Jul 16 17:29:41.884184 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.884120 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" event={"ID":"50bf71f3-1d88-4cb0-ab88-7461907c4fd0","Type":"ContainerStarted","Data":"b3c995bd0cc32c1e281897853207a3e6649c683fd78e3a04c5f3341533c2235e"} Jul 16 17:29:41.885442 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.885413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc" event={"ID":"e915e1a6-38b0-47c0-b9b0-47d4ed2c4809","Type":"ContainerStarted","Data":"ef6d47bf77a8f088d66f0129e8bfea9b6894fe26d4af1b5a96f50de06f2a990e"} Jul 16 17:29:41.886661 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:41.886609 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5" event={"ID":"38a67d8c-f26e-4d5e-b772-2ea4db4b6904","Type":"ContainerStarted","Data":"dfc263dff1108243b07771849398747e06e9a2c624a94b5b89fa784915dd7bbc"} Jul 16 17:29:42.499187 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:42.499151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:42.499187 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:42.499199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:42.499442 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:42.499265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:42.499442 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:42.499311 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:42.499542 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:42.499471 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:44.499456397 +0000 UTC m=+46.251095933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : configmap references non-existent config key: service-ca.crt Jul 16 17:29:42.499613 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:42.499545 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:29:42.499613 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:42.499554 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf54b466-4nz2b: secret "image-registry-tls" not found Jul 16 17:29:42.499613 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:42.499578 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls podName:50a4b64a-aadc-46c6-926d-5148fcd994fd nodeName:}" failed. No retries permitted until 2026-07-16 17:29:44.4995708 +0000 UTC m=+46.251210335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls") pod "image-registry-57cf54b466-4nz2b" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd") : secret "image-registry-tls" not found Jul 16 17:29:42.499765 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:42.499618 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:29:42.499765 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:42.499637 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:44.499631197 +0000 UTC m=+46.251270732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : secret "router-metrics-certs-default" not found Jul 16 17:29:42.499765 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:42.499680 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:42.499765 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:42.499707 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls podName:8debfe7c-6fd2-4ed1-a47b-a916cd09d836 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:44.499699863 +0000 UTC m=+46.251339399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-zvnlg" (UID: "8debfe7c-6fd2-4ed1-a47b-a916cd09d836") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:43.003144 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:43.003104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:43.003594 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:43.003302 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:29:43.003594 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:43.003370 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls podName:ca0ed375-c645-4cc1-9264-67043142deb5 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:51.003354536 +0000 UTC m=+52.754994072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls") pod "dns-default-ztxrd" (UID: "ca0ed375-c645-4cc1-9264-67043142deb5") : secret "dns-default-metrics-tls" not found Jul 16 17:29:44.515644 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:44.515605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:44.515644 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:44.515649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:44.515712 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:44.515737 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:44.515764 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:44.515827 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:44.515844 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls podName:8debfe7c-6fd2-4ed1-a47b-a916cd09d836 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:48.515815538 +0000 UTC m=+50.267455090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-zvnlg" (UID: "8debfe7c-6fd2-4ed1-a47b-a916cd09d836") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:44.515861 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:48.515853294 +0000 UTC m=+50.267492830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : configmap references non-existent config key: service-ca.crt Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:44.515883 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:48.515867245 +0000 UTC m=+50.267506783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : secret "router-metrics-certs-default" not found Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:44.515888 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:44.515907 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf54b466-4nz2b: secret "image-registry-tls" not found Jul 16 17:29:44.516150 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:44.515955 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls podName:50a4b64a-aadc-46c6-926d-5148fcd994fd nodeName:}" failed. No retries permitted until 2026-07-16 17:29:48.515939834 +0000 UTC m=+50.267579375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls") pod "image-registry-57cf54b466-4nz2b" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd") : secret "image-registry-tls" not found Jul 16 17:29:45.897957 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:45.897912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" event={"ID":"50bf71f3-1d88-4cb0-ab88-7461907c4fd0","Type":"ContainerStarted","Data":"c739a7b7de491b063b1da59759eca1c4a08582c151450ef366ecedecb73cb5e7"} Jul 16 17:29:45.899465 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:45.899437 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc" event={"ID":"e915e1a6-38b0-47c0-b9b0-47d4ed2c4809","Type":"ContainerStarted","Data":"832408ea04703b1b391925338331574dff5f9f5f7abebd04e9bc4e87bd01d5cc"} Jul 16 17:29:45.900885 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:45.900857 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5" event={"ID":"38a67d8c-f26e-4d5e-b772-2ea4db4b6904","Type":"ContainerStarted","Data":"5d9967856d838f0867e9906eac0e1df58d92f2598ed75e8faa27aa2a9b585010"} Jul 16 17:29:45.921676 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:45.921631 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" podStartSLOduration=1.8453693260000001 podStartE2EDuration="5.921614214s" totalImagesPullingTime="4.076244888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:45.920932113 +0000 UTC m=+47.672571674" watchObservedRunningTime="2026-07-16 17:29:45.921614214 +0000 UTC m=+47.673253772" Jul 16 17:29:45.943053 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:45.942987 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h9pdc" podStartSLOduration=1.9381340329999999 podStartE2EDuration="5.942965696s" totalImagesPullingTime="4.004831663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:45.941652485 +0000 UTC m=+47.693292044" watchObservedRunningTime="2026-07-16 17:29:45.942965696 +0000 UTC m=+47.694605255" Jul 16 17:29:45.970797 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:45.970734 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gfhb5" podStartSLOduration=2.003986195 podStartE2EDuration="5.970711493s" totalImagesPullingTime="3.966725298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:45.970237725 +0000 UTC m=+47.721877280" watchObservedRunningTime="2026-07-16 17:29:45.970711493 +0000 UTC m=+47.722351054" Jul 16 17:29:46.451944 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.451909 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qpd2g"] Jul 16 17:29:46.481979 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.481946 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.481979 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.481977 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qpd2g"] Jul 16 17:29:46.487109 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.487084 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 17:29:46.487359 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.487335 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 17:29:46.487894 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.487876 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 17:29:46.488640 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.488625 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mkk5\"" Jul 16 17:29:46.488725 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.488625 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 17:29:46.633060 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.633016 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4764c8fb-7014-456c-9ebb-f22b750cb21e-data-volume\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.633261 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.633102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.633261 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.633152 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.633261 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.633230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4764c8fb-7014-456c-9ebb-f22b750cb21e-crio-socket\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.633423 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.633352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrb9\" (UniqueName: \"kubernetes.io/projected/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-api-access-zzrb9\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.734763 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.734658 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.734763 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.734725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.734990 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.734772 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4764c8fb-7014-456c-9ebb-f22b750cb21e-crio-socket\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.734990 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:46.734796 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:29:46.734990 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:46.734862 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:29:47.234848085 +0000 UTC m=+48.986487622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : configmap "kube-rbac-proxy" not found Jul 16 17:29:46.734990 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.734881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrb9\" (UniqueName: \"kubernetes.io/projected/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-api-access-zzrb9\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.734990 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:46.734909 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:29:46.734990 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.734919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4764c8fb-7014-456c-9ebb-f22b750cb21e-data-volume\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.734990 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:46.734969 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:29:47.234952769 +0000 UTC m=+48.986592310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : secret "insights-runtime-extractor-tls" not found Jul 16 17:29:46.735293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.735029 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4764c8fb-7014-456c-9ebb-f22b750cb21e-crio-socket\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.735293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.735273 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4764c8fb-7014-456c-9ebb-f22b750cb21e-data-volume\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:46.749684 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:46.749650 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrb9\" (UniqueName: \"kubernetes.io/projected/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-api-access-zzrb9\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:47.239734 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:47.239702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:47.240140 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:47.239821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:47.240140 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:47.239851 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:29:47.240140 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:47.239891 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:29:47.240140 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:47.239919 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:29:48.239900836 +0000 UTC m=+49.991540371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : secret "insights-runtime-extractor-tls" not found Jul 16 17:29:47.240140 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:47.239938 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:29:48.239926347 +0000 UTC m=+49.991565883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : configmap "kube-rbac-proxy" not found Jul 16 17:29:47.896125 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:47.896097 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c2rnv_42b15b11-e94a-4155-b9b0-1e5f5cf58c92/dns-node-resolver/0.log" Jul 16 17:29:48.249210 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:48.249117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:48.249210 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:48.249177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:48.249670 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.249268 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:29:48.249670 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.249356 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:29:50.249340348 +0000 UTC m=+52.000979884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : configmap "kube-rbac-proxy" not found Jul 16 17:29:48.249670 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.249376 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:29:48.249670 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.249425 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:29:50.249412109 +0000 UTC m=+52.001051650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : secret "insights-runtime-extractor-tls" not found Jul 16 17:29:48.551903 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:48.551870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:48.552081 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:48.551918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:48.552081 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.552034 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:48.552152 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.552096 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:56.552039071 +0000 UTC m=+58.303678624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : configmap references non-existent config key: service-ca.crt Jul 16 17:29:48.552152 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.552129 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls podName:8debfe7c-6fd2-4ed1-a47b-a916cd09d836 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:56.552120134 +0000 UTC m=+58.303759676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-zvnlg" (UID: "8debfe7c-6fd2-4ed1-a47b-a916cd09d836") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:48.552229 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:48.552195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:48.552261 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:48.552226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:48.552361 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.552346 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:29:48.552409 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.552363 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf54b466-4nz2b: secret "image-registry-tls" not found Jul 16 17:29:48.552409 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.552372 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:29:48.552409 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.552403 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:29:56.552391627 +0000 UTC m=+58.304031163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : secret "router-metrics-certs-default" not found Jul 16 17:29:48.552523 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:48.552420 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls podName:50a4b64a-aadc-46c6-926d-5148fcd994fd nodeName:}" failed. No retries permitted until 2026-07-16 17:29:56.552414541 +0000 UTC m=+58.304054077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls") pod "image-registry-57cf54b466-4nz2b" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd") : secret "image-registry-tls" not found Jul 16 17:29:48.902883 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:48.902813 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-88s2w_636d2c81-02e6-4235-a858-d3993e5dba2c/node-ca/0.log" Jul 16 17:29:49.696426 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:49.696399 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6k2d_8cac65f2-474b-4aa7-80d9-a7530cebfbf5/serve-healthcheck-canary/0.log" Jul 16 17:29:50.269161 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:50.269123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:50.269161 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:50.269177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:50.269421 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:50.269266 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:29:50.269421 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:50.269301 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:29:50.269421 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:50.269351 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:29:54.269336791 +0000 UTC m=+56.020976328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : secret "insights-runtime-extractor-tls" not found Jul 16 17:29:50.269421 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:50.269364 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:29:54.269358013 +0000 UTC m=+56.020997549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : configmap "kube-rbac-proxy" not found Jul 16 17:29:51.077978 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:51.077941 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:29:51.078437 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:51.078109 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:29:51.078437 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:51.078173 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls podName:ca0ed375-c645-4cc1-9264-67043142deb5 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:07.078158561 +0000 UTC m=+68.829798097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls") pod "dns-default-ztxrd" (UID: "ca0ed375-c645-4cc1-9264-67043142deb5") : secret "dns-default-metrics-tls" not found Jul 16 17:29:54.303641 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:54.303605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:54.304095 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:54.303734 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:29:54.304095 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:54.303780 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:29:54.304095 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:54.303807 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:29:54.304095 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:54.303863 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:30:02.303849446 +0000 UTC m=+64.055488985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : configmap "kube-rbac-proxy" not found Jul 16 17:29:54.304095 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:54.303881 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:30:02.303872224 +0000 UTC m=+64.055511762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : secret "insights-runtime-extractor-tls" not found Jul 16 17:29:56.625124 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:56.625082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:29:56.625124 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:56.625118 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:56.625165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:56.625182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:56.625244 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:56.625262 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf54b466-4nz2b: secret "image-registry-tls" not found Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:56.625309 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:56.625338 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls podName:50a4b64a-aadc-46c6-926d-5148fcd994fd nodeName:}" failed. No retries permitted until 2026-07-16 17:30:12.62532338 +0000 UTC m=+74.376962916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls") pod "image-registry-57cf54b466-4nz2b" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd") : secret "image-registry-tls" not found Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:56.625309 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:56.625352 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:12.625346193 +0000 UTC m=+74.376985730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : configmap references non-existent config key: service-ca.crt Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:56.625367 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs podName:e4264197-b690-4ca8-a31f-c6c32cf8eba8 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:12.625358268 +0000 UTC m=+74.376997809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs") pod "router-default-6c7798b5fb-gw7rk" (UID: "e4264197-b690-4ca8-a31f-c6c32cf8eba8") : secret "router-metrics-certs-default" not found Jul 16 17:29:56.625665 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:29:56.625379 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls podName:8debfe7c-6fd2-4ed1-a47b-a916cd09d836 nodeName:}" failed. No retries permitted until 2026-07-16 17:30:12.625373474 +0000 UTC m=+74.377013010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-zvnlg" (UID: "8debfe7c-6fd2-4ed1-a47b-a916cd09d836") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:29:58.836379 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:29:58.836345 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-642ls" Jul 16 17:30:02.375912 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:02.375870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:30:02.376316 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:02.375927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:30:02.376316 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:30:02.376095 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:30:02.376316 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:30:02.376180 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls podName:4764c8fb-7014-456c-9ebb-f22b750cb21e nodeName:}" failed. No retries permitted until 2026-07-16 17:30:18.376157593 +0000 UTC m=+80.127797147 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qpd2g" (UID: "4764c8fb-7014-456c-9ebb-f22b750cb21e") : secret "insights-runtime-extractor-tls" not found Jul 16 17:30:02.376555 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:02.376532 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4764c8fb-7014-456c-9ebb-f22b750cb21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:30:04.392878 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.392839 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:30:04.393340 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.392958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:30:04.396623 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.396599 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:30:04.398383 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.398366 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:30:04.406058 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.406038 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d698729-878f-4bb3-a8f6-afd4f2a95ba9-metrics-certs\") pod \"network-metrics-daemon-ns6rx\" (UID: \"4d698729-878f-4bb3-a8f6-afd4f2a95ba9\") " pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:30:04.406149 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.406089 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dd60e07-b295-4af0-b21b-28022c903e65-original-pull-secret\") pod \"global-pull-secret-syncer-f2hc6\" (UID: \"8dd60e07-b295-4af0-b21b-28022c903e65\") " pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:30:04.561357 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.561321 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2hc6" Jul 16 17:30:04.569578 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.569557 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lggz\"" Jul 16 17:30:04.577253 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.577229 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ns6rx" Jul 16 17:30:04.595236 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.595204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:30:04.597754 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.597732 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j252\" (UniqueName: \"kubernetes.io/projected/8e8d5211-4047-48d3-89f6-bb0f78f1775f-kube-api-access-6j252\") pod \"network-check-target-zc7w4\" (UID: \"8e8d5211-4047-48d3-89f6-bb0f78f1775f\") " pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:30:04.721435 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.721356 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2hc6"] Jul 16 17:30:04.723341 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:04.723318 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd60e07_b295_4af0_b21b_28022c903e65.slice/crio-cf135a0c6dc36eeb3907727df06b491f8acb4ccbf1b139b121173ef0f96ae2e1 WatchSource:0}: Error finding container cf135a0c6dc36eeb3907727df06b491f8acb4ccbf1b139b121173ef0f96ae2e1: Status 404 returned error can't find the container with id cf135a0c6dc36eeb3907727df06b491f8acb4ccbf1b139b121173ef0f96ae2e1 Jul 16 17:30:04.739080 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.739058 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ns6rx"] Jul 16 17:30:04.742193 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:04.742167 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d698729_878f_4bb3_a8f6_afd4f2a95ba9.slice/crio-51563d042e2d9456e655c5df9ad7ee9938f0bf981f36f356ccb6f677ded178d2 WatchSource:0}: Error finding container 51563d042e2d9456e655c5df9ad7ee9938f0bf981f36f356ccb6f677ded178d2: Status 404 returned error can't find the container with id 51563d042e2d9456e655c5df9ad7ee9938f0bf981f36f356ccb6f677ded178d2 Jul 16 17:30:04.862778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.862746 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vscvt\"" Jul 16 17:30:04.867866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.867835 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:30:04.951860 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.951820 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2hc6" event={"ID":"8dd60e07-b295-4af0-b21b-28022c903e65","Type":"ContainerStarted","Data":"cf135a0c6dc36eeb3907727df06b491f8acb4ccbf1b139b121173ef0f96ae2e1"} Jul 16 17:30:04.953087 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:04.953054 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ns6rx" event={"ID":"4d698729-878f-4bb3-a8f6-afd4f2a95ba9","Type":"ContainerStarted","Data":"51563d042e2d9456e655c5df9ad7ee9938f0bf981f36f356ccb6f677ded178d2"} Jul 16 17:30:05.008519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:05.008481 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zc7w4"] Jul 16 17:30:05.011700 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:05.011678 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8d5211_4047_48d3_89f6_bb0f78f1775f.slice/crio-06bf91981ea9707926d5f30e03ce8ea280808a1b7bf80adcff78dbbcbf18849f WatchSource:0}: Error finding container 06bf91981ea9707926d5f30e03ce8ea280808a1b7bf80adcff78dbbcbf18849f: Status 404 returned error can't find the container with id 06bf91981ea9707926d5f30e03ce8ea280808a1b7bf80adcff78dbbcbf18849f Jul 16 17:30:05.959304 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:05.959202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zc7w4" event={"ID":"8e8d5211-4047-48d3-89f6-bb0f78f1775f","Type":"ContainerStarted","Data":"323fd7a9b48ff99b7e13d488d0a03f116ac9a000c51dc45ddb4ed7c78145e1fe"} Jul 16 17:30:05.959304 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:05.959252 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zc7w4" event={"ID":"8e8d5211-4047-48d3-89f6-bb0f78f1775f","Type":"ContainerStarted","Data":"06bf91981ea9707926d5f30e03ce8ea280808a1b7bf80adcff78dbbcbf18849f"} Jul 16 17:30:05.959799 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:05.959323 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:30:05.961624 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:05.961596 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ns6rx" event={"ID":"4d698729-878f-4bb3-a8f6-afd4f2a95ba9","Type":"ContainerStarted","Data":"f18063ecbb7f2d12a78b2486577f276b5d5ce91d2fc6d9c7c14fdc798b2583d2"} Jul 16 17:30:05.961736 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:05.961685 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ns6rx" event={"ID":"4d698729-878f-4bb3-a8f6-afd4f2a95ba9","Type":"ContainerStarted","Data":"6a3cc630554b15ca67b430b2fe90f0a0a33aae3391c9fefa3180cfa3dde8e7e1"} Jul 16 17:30:05.989602 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:05.989542 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zc7w4" podStartSLOduration=67.989521382 podStartE2EDuration="1m7.989521382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:05.988314671 +0000 UTC m=+67.739954227" watchObservedRunningTime="2026-07-16 17:30:05.989521382 +0000 UTC m=+67.741160943" Jul 16 17:30:06.027610 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:06.026476 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ns6rx" podStartSLOduration=67.064081556 podStartE2EDuration="1m8.026454165s" totalImagesPullingTime="962.372609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:06.025600196 +0000 UTC m=+67.777239781" watchObservedRunningTime="2026-07-16 17:30:06.026454165 +0000 UTC m=+67.778093725" Jul 16 17:30:07.118340 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:07.118296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:30:07.121374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:07.121343 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca0ed375-c645-4cc1-9264-67043142deb5-metrics-tls\") pod \"dns-default-ztxrd\" (UID: \"ca0ed375-c645-4cc1-9264-67043142deb5\") " pod="openshift-dns/dns-default-ztxrd" Jul 16 17:30:07.344561 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:07.344524 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-27n67\"" Jul 16 17:30:07.352379 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:07.352349 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ztxrd" Jul 16 17:30:07.494404 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:07.494369 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ztxrd"] Jul 16 17:30:07.497835 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:07.497810 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0ed375_c645_4cc1_9264_67043142deb5.slice/crio-aab395a4a3749dc66d394701773800fa2c306afec17b171561370f48812db6d0 WatchSource:0}: Error finding container aab395a4a3749dc66d394701773800fa2c306afec17b171561370f48812db6d0: Status 404 returned error can't find the container with id aab395a4a3749dc66d394701773800fa2c306afec17b171561370f48812db6d0 Jul 16 17:30:07.969441 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:07.969407 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztxrd" event={"ID":"ca0ed375-c645-4cc1-9264-67043142deb5","Type":"ContainerStarted","Data":"aab395a4a3749dc66d394701773800fa2c306afec17b171561370f48812db6d0"} Jul 16 17:30:10.874629 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.874582 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g"] Jul 16 17:30:10.897249 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.897221 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2"] Jul 16 17:30:10.897417 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.897350 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:10.900719 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.900683 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 17:30:10.900719 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.900717 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 17:30:10.900944 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.900799 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 16 17:30:10.901988 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.901970 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 17:30:10.921173 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.921137 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2"] Jul 16 17:30:10.921173 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.921170 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g"] Jul 16 17:30:10.921373 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.921258 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" Jul 16 17:30:10.924687 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.924666 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9b9mp\"" Jul 16 17:30:10.924814 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.924713 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 16 17:30:10.960981 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.960941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f5104708-2d9f-4c2a-9a28-58ab41017577-klusterlet-config\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:10.961131 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.961032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qwz9\" (UniqueName: \"kubernetes.io/projected/f5104708-2d9f-4c2a-9a28-58ab41017577-kube-api-access-5qwz9\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:10.961131 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.961087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5104708-2d9f-4c2a-9a28-58ab41017577-tmp\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:10.980777 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.980746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztxrd" event={"ID":"ca0ed375-c645-4cc1-9264-67043142deb5","Type":"ContainerStarted","Data":"e0e638d09dce41d053c0a03ad09ace47eb18cf050913c47fe55d1ab0459efe32"} Jul 16 17:30:10.980917 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.980799 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztxrd" event={"ID":"ca0ed375-c645-4cc1-9264-67043142deb5","Type":"ContainerStarted","Data":"c720b0084cf7dfb732e83d15bcb1b23c0cad27de2bb2891d4752db48d5665cdf"} Jul 16 17:30:10.980917 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.980862 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ztxrd" Jul 16 17:30:10.982455 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.982432 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2hc6" event={"ID":"8dd60e07-b295-4af0-b21b-28022c903e65","Type":"ContainerStarted","Data":"05c49a9b3d6ccd6534684ed2884838464fa8a3bf526a88b4dd136f18fd897048"} Jul 16 17:30:10.987597 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:10.987577 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-sxqc5"] Jul 16 17:30:11.009026 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.008995 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57cf54b466-4nz2b"] Jul 16 17:30:11.009144 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.009122 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-sxqc5" Jul 16 17:30:11.009191 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:30:11.009174 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" podUID="50a4b64a-aadc-46c6-926d-5148fcd994fd" Jul 16 17:30:11.012128 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.012109 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 17:30:11.012263 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.012134 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 17:30:11.012343 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.012273 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fgfgp\"" Jul 16 17:30:11.016783 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.016762 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-sxqc5"] Jul 16 17:30:11.017227 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.017180 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ztxrd" podStartSLOduration=32.958061131 podStartE2EDuration="36.017162865s" totalImagesPullingTime="3.059101734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:11.015116971 +0000 UTC m=+72.766756533" watchObservedRunningTime="2026-07-16 17:30:11.017162865 +0000 UTC m=+72.768802424" Jul 16 17:30:11.061599 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.061489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19fb3ff3-e1fd-438a-917c-367a606373ae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76d767b744-cjnr2\" (UID: \"19fb3ff3-e1fd-438a-917c-367a606373ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" Jul 16 17:30:11.061599 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.061544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwz9\" (UniqueName: \"kubernetes.io/projected/f5104708-2d9f-4c2a-9a28-58ab41017577-kube-api-access-5qwz9\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:11.061809 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.061623 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5104708-2d9f-4c2a-9a28-58ab41017577-tmp\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:11.061809 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.061669 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqz9\" (UniqueName: \"kubernetes.io/projected/47d72ba2-137d-4c24-a9b2-62f8e2db62fc-kube-api-access-fsqz9\") pod \"downloads-68c9cbb497-sxqc5\" (UID: \"47d72ba2-137d-4c24-a9b2-62f8e2db62fc\") " pod="openshift-console/downloads-68c9cbb497-sxqc5" Jul 16 17:30:11.061809 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.061728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f5104708-2d9f-4c2a-9a28-58ab41017577-klusterlet-config\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:11.061809 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.061786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnctj\" (UniqueName: \"kubernetes.io/projected/19fb3ff3-e1fd-438a-917c-367a606373ae-kube-api-access-bnctj\") pod \"managed-serviceaccount-addon-agent-76d767b744-cjnr2\" (UID: \"19fb3ff3-e1fd-438a-917c-367a606373ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" Jul 16 17:30:11.062488 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.062455 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5104708-2d9f-4c2a-9a28-58ab41017577-tmp\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:11.064826 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.064803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f5104708-2d9f-4c2a-9a28-58ab41017577-klusterlet-config\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:11.088035 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.087979 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f2hc6" podStartSLOduration=67.256984092 podStartE2EDuration="1m13.087963627s" totalImagesPullingTime="5.830979535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:11.085661899 +0000 UTC m=+72.837301468" watchObservedRunningTime="2026-07-16 17:30:11.087963627 +0000 UTC m=+72.839603185" Jul 16 17:30:11.089351 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.089330 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qwz9\" (UniqueName: \"kubernetes.io/projected/f5104708-2d9f-4c2a-9a28-58ab41017577-kube-api-access-5qwz9\") pod \"klusterlet-addon-workmgr-c459f7f7c-xnb7g\" (UID: \"f5104708-2d9f-4c2a-9a28-58ab41017577\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:11.089456 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.089356 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86896dd949-lcm9g"] Jul 16 17:30:11.113103 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.113075 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86896dd949-lcm9g"] Jul 16 17:30:11.113232 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.113160 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.163096 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.163062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnctj\" (UniqueName: \"kubernetes.io/projected/19fb3ff3-e1fd-438a-917c-367a606373ae-kube-api-access-bnctj\") pod \"managed-serviceaccount-addon-agent-76d767b744-cjnr2\" (UID: \"19fb3ff3-e1fd-438a-917c-367a606373ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" Jul 16 17:30:11.163272 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.163107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19fb3ff3-e1fd-438a-917c-367a606373ae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76d767b744-cjnr2\" (UID: \"19fb3ff3-e1fd-438a-917c-367a606373ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" Jul 16 17:30:11.163272 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.163169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqz9\" (UniqueName: \"kubernetes.io/projected/47d72ba2-137d-4c24-a9b2-62f8e2db62fc-kube-api-access-fsqz9\") pod \"downloads-68c9cbb497-sxqc5\" (UID: \"47d72ba2-137d-4c24-a9b2-62f8e2db62fc\") " pod="openshift-console/downloads-68c9cbb497-sxqc5" Jul 16 17:30:11.165766 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.165746 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/19fb3ff3-e1fd-438a-917c-367a606373ae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76d767b744-cjnr2\" (UID: \"19fb3ff3-e1fd-438a-917c-367a606373ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" Jul 16 17:30:11.177672 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.177641 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnctj\" (UniqueName: \"kubernetes.io/projected/19fb3ff3-e1fd-438a-917c-367a606373ae-kube-api-access-bnctj\") pod \"managed-serviceaccount-addon-agent-76d767b744-cjnr2\" (UID: \"19fb3ff3-e1fd-438a-917c-367a606373ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" Jul 16 17:30:11.177778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.177768 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqz9\" (UniqueName: \"kubernetes.io/projected/47d72ba2-137d-4c24-a9b2-62f8e2db62fc-kube-api-access-fsqz9\") pod \"downloads-68c9cbb497-sxqc5\" (UID: \"47d72ba2-137d-4c24-a9b2-62f8e2db62fc\") " pod="openshift-console/downloads-68c9cbb497-sxqc5" Jul 16 17:30:11.207045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.207023 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:11.246726 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.246702 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" Jul 16 17:30:11.264607 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.264577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-registry-tls\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.264772 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.264617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2dd999b4-ca4a-4082-b79b-f31a5700b625-trusted-ca\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.264772 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.264651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2dd999b4-ca4a-4082-b79b-f31a5700b625-registry-certificates\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.264878 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.264776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2dd999b4-ca4a-4082-b79b-f31a5700b625-ca-trust-extracted\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.264878 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.264815 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2dd999b4-ca4a-4082-b79b-f31a5700b625-installation-pull-secrets\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.264952 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.264880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2dd999b4-ca4a-4082-b79b-f31a5700b625-image-registry-private-configuration\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.264952 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.264911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-bound-sa-token\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.264952 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.264931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scnwg\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-kube-api-access-scnwg\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.319117 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.319089 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-sxqc5" Jul 16 17:30:11.366254 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.366222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-registry-tls\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.366429 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.366269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2dd999b4-ca4a-4082-b79b-f31a5700b625-trusted-ca\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.366429 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.366345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2dd999b4-ca4a-4082-b79b-f31a5700b625-registry-certificates\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.366531 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.366436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2dd999b4-ca4a-4082-b79b-f31a5700b625-ca-trust-extracted\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.366531 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.366470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2dd999b4-ca4a-4082-b79b-f31a5700b625-installation-pull-secrets\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.366630 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.366539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2dd999b4-ca4a-4082-b79b-f31a5700b625-image-registry-private-configuration\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.366630 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.366587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-bound-sa-token\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.366729 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.366625 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scnwg\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-kube-api-access-scnwg\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.367407 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.367356 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2dd999b4-ca4a-4082-b79b-f31a5700b625-ca-trust-extracted\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.367673 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.367612 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2dd999b4-ca4a-4082-b79b-f31a5700b625-registry-certificates\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.369241 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.369213 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2dd999b4-ca4a-4082-b79b-f31a5700b625-trusted-ca\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.371819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.371781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2dd999b4-ca4a-4082-b79b-f31a5700b625-image-registry-private-configuration\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.371930 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.371890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2dd999b4-ca4a-4082-b79b-f31a5700b625-installation-pull-secrets\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.372240 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.372171 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-registry-tls\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.381646 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.381561 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g"] Jul 16 17:30:11.384888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.384246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-bound-sa-token\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.385441 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.385369 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scnwg\" (UniqueName: \"kubernetes.io/projected/2dd999b4-ca4a-4082-b79b-f31a5700b625-kube-api-access-scnwg\") pod \"image-registry-86896dd949-lcm9g\" (UID: \"2dd999b4-ca4a-4082-b79b-f31a5700b625\") " pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.424604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.424573 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wm2ng\"" Jul 16 17:30:11.427980 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.427950 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2"] Jul 16 17:30:11.430737 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:11.430704 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19fb3ff3_e1fd_438a_917c_367a606373ae.slice/crio-1059d5ccf6d5517e5f7471e7ee61c7d5f56472951a31f86814726e616f4b409c WatchSource:0}: Error finding container 1059d5ccf6d5517e5f7471e7ee61c7d5f56472951a31f86814726e616f4b409c: Status 404 returned error can't find the container with id 1059d5ccf6d5517e5f7471e7ee61c7d5f56472951a31f86814726e616f4b409c Jul 16 17:30:11.431711 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.431693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.490384 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.490198 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-sxqc5"] Jul 16 17:30:11.493096 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:11.493064 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d72ba2_137d_4c24_a9b2_62f8e2db62fc.slice/crio-687a9d2627bb567196683031093d1cfff72a13a9f71386abdc2502ee5051c4c2 WatchSource:0}: Error finding container 687a9d2627bb567196683031093d1cfff72a13a9f71386abdc2502ee5051c4c2: Status 404 returned error can't find the container with id 687a9d2627bb567196683031093d1cfff72a13a9f71386abdc2502ee5051c4c2 Jul 16 17:30:11.596298 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.596038 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86896dd949-lcm9g"] Jul 16 17:30:11.598420 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:11.598391 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd999b4_ca4a_4082_b79b_f31a5700b625.slice/crio-dddf2e1827d0fa147492ca6587f6cf78c322c870eddaa306df64c82219746bae WatchSource:0}: Error finding container dddf2e1827d0fa147492ca6587f6cf78c322c870eddaa306df64c82219746bae: Status 404 returned error can't find the container with id dddf2e1827d0fa147492ca6587f6cf78c322c870eddaa306df64c82219746bae Jul 16 17:30:11.988337 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.988229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86896dd949-lcm9g" event={"ID":"2dd999b4-ca4a-4082-b79b-f31a5700b625","Type":"ContainerStarted","Data":"ae65b3d61dd7067cd305b77feb6e0857932c1f2a38deb850e71bb202fc221ab2"} Jul 16 17:30:11.988337 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.988300 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86896dd949-lcm9g" event={"ID":"2dd999b4-ca4a-4082-b79b-f31a5700b625","Type":"ContainerStarted","Data":"dddf2e1827d0fa147492ca6587f6cf78c322c870eddaa306df64c82219746bae"} Jul 16 17:30:11.988777 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.988342 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:11.989580 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.989554 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" event={"ID":"19fb3ff3-e1fd-438a-917c-367a606373ae","Type":"ContainerStarted","Data":"1059d5ccf6d5517e5f7471e7ee61c7d5f56472951a31f86814726e616f4b409c"} Jul 16 17:30:11.990685 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.990661 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-sxqc5" event={"ID":"47d72ba2-137d-4c24-a9b2-62f8e2db62fc","Type":"ContainerStarted","Data":"687a9d2627bb567196683031093d1cfff72a13a9f71386abdc2502ee5051c4c2"} Jul 16 17:30:11.991701 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.991687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:30:11.991790 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:11.991741 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" event={"ID":"f5104708-2d9f-4c2a-9a28-58ab41017577","Type":"ContainerStarted","Data":"1bdad6693516674846786bc37403ba6a56666f33fa23db07b0fb9a1a9ba10e69"} Jul 16 17:30:12.011796 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.011777 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:30:12.017063 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.017020 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86896dd949-lcm9g" podStartSLOduration=1.017005402 podStartE2EDuration="1.017005402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:12.016062946 +0000 UTC m=+73.767702530" watchObservedRunningTime="2026-07-16 17:30:12.017005402 +0000 UTC m=+73.768644937" Jul 16 17:30:12.073232 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073197 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50a4b64a-aadc-46c6-926d-5148fcd994fd-ca-trust-extracted\") pod \"50a4b64a-aadc-46c6-926d-5148fcd994fd\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " Jul 16 17:30:12.073437 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073255 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-installation-pull-secrets\") pod \"50a4b64a-aadc-46c6-926d-5148fcd994fd\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " Jul 16 17:30:12.073437 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073305 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thmz8\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-kube-api-access-thmz8\") pod \"50a4b64a-aadc-46c6-926d-5148fcd994fd\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " Jul 16 17:30:12.073437 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073331 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-bound-sa-token\") pod \"50a4b64a-aadc-46c6-926d-5148fcd994fd\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " Jul 16 17:30:12.073437 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073357 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-image-registry-private-configuration\") pod \"50a4b64a-aadc-46c6-926d-5148fcd994fd\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " Jul 16 17:30:12.073437 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073382 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-certificates\") pod \"50a4b64a-aadc-46c6-926d-5148fcd994fd\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " Jul 16 17:30:12.073437 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073409 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-trusted-ca\") pod \"50a4b64a-aadc-46c6-926d-5148fcd994fd\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " Jul 16 17:30:12.073721 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073542 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a4b64a-aadc-46c6-926d-5148fcd994fd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "50a4b64a-aadc-46c6-926d-5148fcd994fd" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:30:12.073851 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.073830 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50a4b64a-aadc-46c6-926d-5148fcd994fd-ca-trust-extracted\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:30:12.074249 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.074225 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "50a4b64a-aadc-46c6-926d-5148fcd994fd" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:30:12.074551 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.074526 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "50a4b64a-aadc-46c6-926d-5148fcd994fd" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:30:12.076165 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.076139 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "50a4b64a-aadc-46c6-926d-5148fcd994fd" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:30:12.076307 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.076179 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-kube-api-access-thmz8" (OuterVolumeSpecName: "kube-api-access-thmz8") pod "50a4b64a-aadc-46c6-926d-5148fcd994fd" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd"). InnerVolumeSpecName "kube-api-access-thmz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:30:12.076307 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.076249 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "50a4b64a-aadc-46c6-926d-5148fcd994fd" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:30:12.076981 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.076963 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "50a4b64a-aadc-46c6-926d-5148fcd994fd" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:30:12.174249 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.174215 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-installation-pull-secrets\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:30:12.174249 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.174247 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmz8\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-kube-api-access-thmz8\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:30:12.174249 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.174259 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-bound-sa-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:30:12.174514 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.174271 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50a4b64a-aadc-46c6-926d-5148fcd994fd-image-registry-private-configuration\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:30:12.174514 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.174304 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-certificates\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:30:12.174514 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.174316 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50a4b64a-aadc-46c6-926d-5148fcd994fd-trusted-ca\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:30:12.679137 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.679101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:30:12.679315 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.679152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:12.679315 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.679211 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:30:12.679315 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.679243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:12.680039 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.679992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4264197-b690-4ca8-a31f-c6c32cf8eba8-service-ca-bundle\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:12.682374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.682326 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4264197-b690-4ca8-a31f-c6c32cf8eba8-metrics-certs\") pod \"router-default-6c7798b5fb-gw7rk\" (UID: \"e4264197-b690-4ca8-a31f-c6c32cf8eba8\") " pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:12.682374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.682351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"image-registry-57cf54b466-4nz2b\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:30:12.682731 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.682711 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8debfe7c-6fd2-4ed1-a47b-a916cd09d836-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-zvnlg\" (UID: \"8debfe7c-6fd2-4ed1-a47b-a916cd09d836\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:30:12.780542 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.779922 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") pod \"50a4b64a-aadc-46c6-926d-5148fcd994fd\" (UID: \"50a4b64a-aadc-46c6-926d-5148fcd994fd\") " Jul 16 17:30:12.783734 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.783687 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "50a4b64a-aadc-46c6-926d-5148fcd994fd" (UID: "50a4b64a-aadc-46c6-926d-5148fcd994fd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:30:12.881009 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.880975 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50a4b64a-aadc-46c6-926d-5148fcd994fd-registry-tls\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:30:12.881567 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.881270 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2j945\"" Jul 16 17:30:12.889207 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.888910 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" Jul 16 17:30:12.901173 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.900968 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nc8wp\"" Jul 16 17:30:12.909523 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:12.909451 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:13.001336 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:13.000337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cf54b466-4nz2b" Jul 16 17:30:13.059541 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:13.059308 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57cf54b466-4nz2b"] Jul 16 17:30:13.061556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:13.061523 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57cf54b466-4nz2b"] Jul 16 17:30:13.086692 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:13.086654 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg"] Jul 16 17:30:13.090924 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:13.090894 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8debfe7c_6fd2_4ed1_a47b_a916cd09d836.slice/crio-1d15df9f0323827b919f8ae9c4471ab49ea18633f7a10d85d20ac566f3c743dc WatchSource:0}: Error finding container 1d15df9f0323827b919f8ae9c4471ab49ea18633f7a10d85d20ac566f3c743dc: Status 404 returned error can't find the container with id 1d15df9f0323827b919f8ae9c4471ab49ea18633f7a10d85d20ac566f3c743dc Jul 16 17:30:13.132162 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:13.132116 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c7798b5fb-gw7rk"] Jul 16 17:30:14.019553 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:14.019511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" event={"ID":"8debfe7c-6fd2-4ed1-a47b-a916cd09d836","Type":"ContainerStarted","Data":"1d15df9f0323827b919f8ae9c4471ab49ea18633f7a10d85d20ac566f3c743dc"} Jul 16 17:30:14.022876 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:14.022841 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4264197_b690_4ca8_a31f_c6c32cf8eba8.slice/crio-8c1143f67e9120d2ca9038e944c31ec044bfcb9b78c0383d1340a0b9c557dd24 WatchSource:0}: Error finding container 8c1143f67e9120d2ca9038e944c31ec044bfcb9b78c0383d1340a0b9c557dd24: Status 404 returned error can't find the container with id 8c1143f67e9120d2ca9038e944c31ec044bfcb9b78c0383d1340a0b9c557dd24 Jul 16 17:30:14.650691 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:14.650659 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a4b64a-aadc-46c6-926d-5148fcd994fd" path="/var/lib/kubelet/pods/50a4b64a-aadc-46c6-926d-5148fcd994fd/volumes" Jul 16 17:30:15.025558 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:15.025461 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" event={"ID":"19fb3ff3-e1fd-438a-917c-367a606373ae","Type":"ContainerStarted","Data":"0260799189a4edb97e28c2d30b462772cfaf196850c9ea0b5e4c639c627beac9"} Jul 16 17:30:15.028092 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:15.028044 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" event={"ID":"e4264197-b690-4ca8-a31f-c6c32cf8eba8","Type":"ContainerStarted","Data":"260a2220ca45a38fa1fd1b55231cc2ec011b1d9cfe671e21dfbe0114bcf5791a"} Jul 16 17:30:15.028238 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:15.028100 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" event={"ID":"e4264197-b690-4ca8-a31f-c6c32cf8eba8","Type":"ContainerStarted","Data":"8c1143f67e9120d2ca9038e944c31ec044bfcb9b78c0383d1340a0b9c557dd24"} Jul 16 17:30:15.048060 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:15.047988 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76d767b744-cjnr2" podStartSLOduration=1.702835888 podStartE2EDuration="5.047972789s" totalImagesPullingTime="3.345136901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:15.047319784 +0000 UTC m=+76.798959342" watchObservedRunningTime="2026-07-16 17:30:15.047972789 +0000 UTC m=+76.799612325" Jul 16 17:30:15.079004 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:15.078949 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" podStartSLOduration=35.078932864 podStartE2EDuration="35.078932864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:15.078389421 +0000 UTC m=+76.830029001" watchObservedRunningTime="2026-07-16 17:30:15.078932864 +0000 UTC m=+76.830572421" Jul 16 17:30:15.909724 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:15.909681 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:15.912587 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:15.912552 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:16.033304 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:16.033256 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:16.034835 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:16.034809 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c7798b5fb-gw7rk" Jul 16 17:30:17.038497 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:17.038460 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" event={"ID":"8debfe7c-6fd2-4ed1-a47b-a916cd09d836","Type":"ContainerStarted","Data":"eeed853f80985d2ad87f2e92e7f35430fb48bfbf946ee797e9fffdea56190e9c"} Jul 16 17:30:17.070523 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:17.070476 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-zvnlg" podStartSLOduration=33.918262632 podStartE2EDuration="37.070458338s" totalImagesPullingTime="3.152195706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:17.068437615 +0000 UTC m=+78.820077172" watchObservedRunningTime="2026-07-16 17:30:17.070458338 +0000 UTC m=+78.822097929" Jul 16 17:30:18.432801 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:18.432760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:30:18.435172 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:18.435139 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4764c8fb-7014-456c-9ebb-f22b750cb21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qpd2g\" (UID: \"4764c8fb-7014-456c-9ebb-f22b750cb21e\") " pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:30:18.596947 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:18.596913 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mkk5\"" Jul 16 17:30:18.603835 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:18.603807 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qpd2g" Jul 16 17:30:18.773107 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:18.773076 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qpd2g"] Jul 16 17:30:18.778339 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:18.778307 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4764c8fb_7014_456c_9ebb_f22b750cb21e.slice/crio-e5303fd6d1529e9a553a7149bf2fa4987ec7b9adef44fa5feeb5d38a85653fbd WatchSource:0}: Error finding container e5303fd6d1529e9a553a7149bf2fa4987ec7b9adef44fa5feeb5d38a85653fbd: Status 404 returned error can't find the container with id e5303fd6d1529e9a553a7149bf2fa4987ec7b9adef44fa5feeb5d38a85653fbd Jul 16 17:30:19.048556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:19.048515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qpd2g" event={"ID":"4764c8fb-7014-456c-9ebb-f22b750cb21e","Type":"ContainerStarted","Data":"e5303fd6d1529e9a553a7149bf2fa4987ec7b9adef44fa5feeb5d38a85653fbd"} Jul 16 17:30:20.994119 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:20.994092 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ztxrd" Jul 16 17:30:31.596399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.596365 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tdfgm"] Jul 16 17:30:31.617573 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.617545 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.624227 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.624197 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 17:30:31.631100 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.631075 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 17:30:31.631247 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.631075 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nmknv\"" Jul 16 17:30:31.631247 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.631075 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 17:30:31.631247 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.631119 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 17:30:31.748372 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj97s\" (UniqueName: \"kubernetes.io/projected/35316ded-1df4-4993-8257-cb72b0ed749f-kube-api-access-dj97s\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.748372 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-sys\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.748631 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748403 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-accelerators-collector-config\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.748631 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35316ded-1df4-4993-8257-cb72b0ed749f-metrics-client-ca\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.748631 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-root\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.748631 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-textfile\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.748631 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.748839 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-tls\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.748839 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.748682 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-wtmp\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849332 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-root\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849332 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-textfile\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-tls\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849362 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-root\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-wtmp\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj97s\" (UniqueName: \"kubernetes.io/projected/35316ded-1df4-4993-8257-cb72b0ed749f-kube-api-access-dj97s\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-sys\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-accelerators-collector-config\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849508 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-wtmp\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35316ded-1df4-4993-8257-cb72b0ed749f-metrics-client-ca\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849853 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849575 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35316ded-1df4-4993-8257-cb72b0ed749f-sys\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.849853 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.849729 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-textfile\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.850063 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.850029 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35316ded-1df4-4993-8257-cb72b0ed749f-metrics-client-ca\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.850063 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.850029 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-accelerators-collector-config\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.851801 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.851781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-tls\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.851902 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.851822 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35316ded-1df4-4993-8257-cb72b0ed749f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.863194 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.863171 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj97s\" (UniqueName: \"kubernetes.io/projected/35316ded-1df4-4993-8257-cb72b0ed749f-kube-api-access-dj97s\") pod \"node-exporter-tdfgm\" (UID: \"35316ded-1df4-4993-8257-cb72b0ed749f\") " pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:31.928124 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:31.928084 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tdfgm" Jul 16 17:30:33.006100 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:33.006069 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86896dd949-lcm9g" Jul 16 17:30:33.955094 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:30:33.955061 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35316ded_1df4_4993_8257_cb72b0ed749f.slice/crio-60046f13251940eed32b5e2a8e762a9aae2bf6f5d59be1120efcaaaa71072a3e WatchSource:0}: Error finding container 60046f13251940eed32b5e2a8e762a9aae2bf6f5d59be1120efcaaaa71072a3e: Status 404 returned error can't find the container with id 60046f13251940eed32b5e2a8e762a9aae2bf6f5d59be1120efcaaaa71072a3e Jul 16 17:30:34.115000 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.114961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" event={"ID":"f5104708-2d9f-4c2a-9a28-58ab41017577","Type":"ContainerStarted","Data":"02c1db2d7bf07e3ad12bacf878f9946363598702f8f795412aeb970f4f661ea2"} Jul 16 17:30:34.115490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.115156 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:34.117044 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.117010 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qpd2g" event={"ID":"4764c8fb-7014-456c-9ebb-f22b750cb21e","Type":"ContainerStarted","Data":"397b8229bb362c4c85e8b1657d9058aae79fff95352ccc84bf58ff74693d5ab3"} Jul 16 17:30:34.117175 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.117090 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" Jul 16 17:30:34.118480 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.118451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-sxqc5" event={"ID":"47d72ba2-137d-4c24-a9b2-62f8e2db62fc","Type":"ContainerStarted","Data":"957c64ee80eb6b6d19b01990c0ed193911316dd581469581e7b905fb86d641e3"} Jul 16 17:30:34.118872 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.118841 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-sxqc5" Jul 16 17:30:34.120547 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.120516 2582 patch_prober.go:28] interesting pod/downloads-68c9cbb497-sxqc5 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.17:8080/\": dial tcp 10.133.0.17:8080: connect: connection refused" start-of-body= Jul 16 17:30:34.120654 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.120557 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdfgm" event={"ID":"35316ded-1df4-4993-8257-cb72b0ed749f","Type":"ContainerStarted","Data":"60046f13251940eed32b5e2a8e762a9aae2bf6f5d59be1120efcaaaa71072a3e"} Jul 16 17:30:34.120654 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.120576 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-sxqc5" podUID="47d72ba2-137d-4c24-a9b2-62f8e2db62fc" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.17:8080/\": dial tcp 10.133.0.17:8080: connect: connection refused" Jul 16 17:30:34.146431 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.146369 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c459f7f7c-xnb7g" podStartSLOduration=1.571351565 podStartE2EDuration="24.14635048s" totalImagesPullingTime="22.574998915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:34.145373327 +0000 UTC m=+95.897012910" watchObservedRunningTime="2026-07-16 17:30:34.14635048 +0000 UTC m=+95.897990040" Jul 16 17:30:34.222978 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:34.222865 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-sxqc5" podStartSLOduration=1.766147063 podStartE2EDuration="24.222852192s" totalImagesPullingTime="22.456705129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:34.183368872 +0000 UTC m=+95.935008447" watchObservedRunningTime="2026-07-16 17:30:34.222852192 +0000 UTC m=+95.974491784" Jul 16 17:30:35.139106 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:35.139069 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-sxqc5" Jul 16 17:30:36.967695 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:36.967661 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zc7w4" Jul 16 17:30:40.149467 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:40.149388 2582 generic.go:356] "Generic (PLEG): container finished" podID="35316ded-1df4-4993-8257-cb72b0ed749f" containerID="b21bddb59ffef52424dfb2519d154097c1423ba849a9de54552a6172f2c977bf" exitCode=0 Jul 16 17:30:40.149904 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:40.149521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdfgm" event={"ID":"35316ded-1df4-4993-8257-cb72b0ed749f","Type":"ContainerDied","Data":"b21bddb59ffef52424dfb2519d154097c1423ba849a9de54552a6172f2c977bf"} Jul 16 17:30:40.152256 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:40.152227 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qpd2g" event={"ID":"4764c8fb-7014-456c-9ebb-f22b750cb21e","Type":"ContainerStarted","Data":"630d56d34e8412e798e33b60fd9cddbf0b7da6a05a58bbd770a5fdfc4ce9fe8c"} Jul 16 17:30:41.160314 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:41.159506 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdfgm" event={"ID":"35316ded-1df4-4993-8257-cb72b0ed749f","Type":"ContainerStarted","Data":"4d7e8d23f59dda2e2e3b3a5d6c56b8137499b3d9ba37c5c8d722bd2d234260ab"} Jul 16 17:30:41.160314 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:41.159574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdfgm" event={"ID":"35316ded-1df4-4993-8257-cb72b0ed749f","Type":"ContainerStarted","Data":"bbf832b4b82aa2a91cfd3132dc51658e5dd9400a19aeac9b2381b9f461e7e463"} Jul 16 17:30:41.185614 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:41.185547 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tdfgm" podStartSLOduration=4.395786472 podStartE2EDuration="10.185522521s" totalImagesPullingTime="5.666304645s" totalInitContainerRuntime="123.431404ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:41.182979856 +0000 UTC m=+102.934619438" watchObservedRunningTime="2026-07-16 17:30:41.185522521 +0000 UTC m=+102.937162080" Jul 16 17:30:45.177509 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:45.177415 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qpd2g" event={"ID":"4764c8fb-7014-456c-9ebb-f22b750cb21e","Type":"ContainerStarted","Data":"7880ce542c47f4af4212be81ba723388987b673128514d7e23355ee1c95bb482"} Jul 16 17:30:45.201756 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:30:45.201615 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qpd2g" podStartSLOduration=33.326656042 podStartE2EDuration="59.201591804s" totalImagesPullingTime="25.874935762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:29:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:45.20052384 +0000 UTC m=+106.952163409" watchObservedRunningTime="2026-07-16 17:30:45.201591804 +0000 UTC m=+106.953231363" Jul 16 17:31:17.295255 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:31:17.295227 2582 generic.go:356] "Generic (PLEG): container finished" podID="50bf71f3-1d88-4cb0-ab88-7461907c4fd0" containerID="c739a7b7de491b063b1da59759eca1c4a08582c151450ef366ecedecb73cb5e7" exitCode=0 Jul 16 17:31:17.295656 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:31:17.295306 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" event={"ID":"50bf71f3-1d88-4cb0-ab88-7461907c4fd0","Type":"ContainerDied","Data":"c739a7b7de491b063b1da59759eca1c4a08582c151450ef366ecedecb73cb5e7"} Jul 16 17:31:17.295656 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:31:17.295629 2582 scope.go:117] "RemoveContainer" containerID="c739a7b7de491b063b1da59759eca1c4a08582c151450ef366ecedecb73cb5e7" Jul 16 17:31:18.300638 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:31:18.300604 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qh8vr" event={"ID":"50bf71f3-1d88-4cb0-ab88-7461907c4fd0","Type":"ContainerStarted","Data":"5bf505997ce9b8d581dcf8ad5d528fe195644913fc3af028d345de0269cd967d"} Jul 16 17:33:58.573585 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:33:58.573557 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 17:35:04.212990 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.212902 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn"] Jul 16 17:35:04.216297 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.216259 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.219980 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.219964 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 17:35:04.220099 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.220022 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mzvtg\"" Jul 16 17:35:04.221238 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.221221 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 17:35:04.230151 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.230123 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn"] Jul 16 17:35:04.373257 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.373222 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.373447 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.373275 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.373447 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.373349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5t76\" (UniqueName: \"kubernetes.io/projected/425689cf-5067-4390-937c-b3d60903b293-kube-api-access-m5t76\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.474225 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.474143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.474225 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.474185 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.474225 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.474216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5t76\" (UniqueName: \"kubernetes.io/projected/425689cf-5067-4390-937c-b3d60903b293-kube-api-access-m5t76\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.474540 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.474523 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.474622 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.474605 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.490754 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.490726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5t76\" (UniqueName: \"kubernetes.io/projected/425689cf-5067-4390-937c-b3d60903b293-kube-api-access-m5t76\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.524793 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.524763 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:04.669888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.669859 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn"] Jul 16 17:35:04.673045 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:35:04.673011 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425689cf_5067_4390_937c_b3d60903b293.slice/crio-e08f61f06c6d16f822aafd57e61f83933d381d02abb6ecbf4190fdca6984f397 WatchSource:0}: Error finding container e08f61f06c6d16f822aafd57e61f83933d381d02abb6ecbf4190fdca6984f397: Status 404 returned error can't find the container with id e08f61f06c6d16f822aafd57e61f83933d381d02abb6ecbf4190fdca6984f397 Jul 16 17:35:04.675305 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:04.675269 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:35:05.066703 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:05.066671 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" event={"ID":"425689cf-5067-4390-937c-b3d60903b293","Type":"ContainerStarted","Data":"e08f61f06c6d16f822aafd57e61f83933d381d02abb6ecbf4190fdca6984f397"} Jul 16 17:35:13.097025 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:13.096983 2582 generic.go:356] "Generic (PLEG): container finished" podID="425689cf-5067-4390-937c-b3d60903b293" containerID="e0b5b21df449cb7bc36f34db6309d6ac2cd34375bb251a74c7966f52f285ce3d" exitCode=0 Jul 16 17:35:13.097442 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:13.097074 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" event={"ID":"425689cf-5067-4390-937c-b3d60903b293","Type":"ContainerDied","Data":"e0b5b21df449cb7bc36f34db6309d6ac2cd34375bb251a74c7966f52f285ce3d"} Jul 16 17:35:16.110265 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:16.110229 2582 generic.go:356] "Generic (PLEG): container finished" podID="425689cf-5067-4390-937c-b3d60903b293" containerID="c66a5a92315b20f3caab4fa034bb56815e9779e8cd9d9c5dc3abebb372a5483c" exitCode=0 Jul 16 17:35:16.110265 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:16.110269 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" event={"ID":"425689cf-5067-4390-937c-b3d60903b293","Type":"ContainerDied","Data":"c66a5a92315b20f3caab4fa034bb56815e9779e8cd9d9c5dc3abebb372a5483c"} Jul 16 17:35:25.147128 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:25.147094 2582 generic.go:356] "Generic (PLEG): container finished" podID="425689cf-5067-4390-937c-b3d60903b293" containerID="0f31a951c396f8faa8bc25992bf0d71cd5e02906ea589c3a170b3832f59b8d8c" exitCode=0 Jul 16 17:35:25.147513 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:25.147187 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" event={"ID":"425689cf-5067-4390-937c-b3d60903b293","Type":"ContainerDied","Data":"0f31a951c396f8faa8bc25992bf0d71cd5e02906ea589c3a170b3832f59b8d8c"} Jul 16 17:35:26.325913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.325880 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:26.443577 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.443502 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-util\") pod \"425689cf-5067-4390-937c-b3d60903b293\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " Jul 16 17:35:26.443577 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.443558 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5t76\" (UniqueName: \"kubernetes.io/projected/425689cf-5067-4390-937c-b3d60903b293-kube-api-access-m5t76\") pod \"425689cf-5067-4390-937c-b3d60903b293\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " Jul 16 17:35:26.443757 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.443592 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-bundle\") pod \"425689cf-5067-4390-937c-b3d60903b293\" (UID: \"425689cf-5067-4390-937c-b3d60903b293\") " Jul 16 17:35:26.444416 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.444385 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-bundle" (OuterVolumeSpecName: "bundle") pod "425689cf-5067-4390-937c-b3d60903b293" (UID: "425689cf-5067-4390-937c-b3d60903b293"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:35:26.445898 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.445875 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425689cf-5067-4390-937c-b3d60903b293-kube-api-access-m5t76" (OuterVolumeSpecName: "kube-api-access-m5t76") pod "425689cf-5067-4390-937c-b3d60903b293" (UID: "425689cf-5067-4390-937c-b3d60903b293"). InnerVolumeSpecName "kube-api-access-m5t76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:35:26.448022 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.448003 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-util" (OuterVolumeSpecName: "util") pod "425689cf-5067-4390-937c-b3d60903b293" (UID: "425689cf-5067-4390-937c-b3d60903b293"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:35:26.544374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.544339 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5t76\" (UniqueName: \"kubernetes.io/projected/425689cf-5067-4390-937c-b3d60903b293-kube-api-access-m5t76\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:35:26.544374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.544366 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-bundle\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:35:26.544374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:26.544376 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425689cf-5067-4390-937c-b3d60903b293-util\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:35:27.155493 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:27.155462 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" Jul 16 17:35:27.155634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:27.155467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vqgn" event={"ID":"425689cf-5067-4390-937c-b3d60903b293","Type":"ContainerDied","Data":"e08f61f06c6d16f822aafd57e61f83933d381d02abb6ecbf4190fdca6984f397"} Jul 16 17:35:27.155634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:27.155587 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e08f61f06c6d16f822aafd57e61f83933d381d02abb6ecbf4190fdca6984f397" Jul 16 17:35:38.270270 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.270227 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn"] Jul 16 17:35:38.270724 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.270631 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425689cf-5067-4390-937c-b3d60903b293" containerName="pull" Jul 16 17:35:38.270724 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.270648 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="425689cf-5067-4390-937c-b3d60903b293" containerName="pull" Jul 16 17:35:38.270724 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.270672 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425689cf-5067-4390-937c-b3d60903b293" containerName="util" Jul 16 17:35:38.270724 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.270680 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="425689cf-5067-4390-937c-b3d60903b293" containerName="util" Jul 16 17:35:38.270724 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.270692 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425689cf-5067-4390-937c-b3d60903b293" containerName="extract" Jul 16 17:35:38.270724 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.270699 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="425689cf-5067-4390-937c-b3d60903b293" containerName="extract" Jul 16 17:35:38.271009 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.270781 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="425689cf-5067-4390-937c-b3d60903b293" containerName="extract" Jul 16 17:35:38.273646 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.273624 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.281903 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.281876 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mzvtg\"" Jul 16 17:35:38.282004 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.281887 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 17:35:38.282911 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.282897 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 17:35:38.295396 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.295368 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn"] Jul 16 17:35:38.431878 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.431843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.432054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.431883 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.432054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.431911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2w52\" (UniqueName: \"kubernetes.io/projected/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-kube-api-access-v2w52\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.532663 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.532636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.532820 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.532672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.532820 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.532695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2w52\" (UniqueName: \"kubernetes.io/projected/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-kube-api-access-v2w52\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.533030 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.533005 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.533079 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.533034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.556750 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.556714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2w52\" (UniqueName: \"kubernetes.io/projected/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-kube-api-access-v2w52\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.582509 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.582476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:38.756177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:38.756152 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn"] Jul 16 17:35:38.758419 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:35:38.758393 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275d0233_fb85_4457_bbaf_5a6d5c8c39c4.slice/crio-879515fecf24337c977fba1afa141b1f3c2ab4fb8019b290fec258b916a68320 WatchSource:0}: Error finding container 879515fecf24337c977fba1afa141b1f3c2ab4fb8019b290fec258b916a68320: Status 404 returned error can't find the container with id 879515fecf24337c977fba1afa141b1f3c2ab4fb8019b290fec258b916a68320 Jul 16 17:35:39.202530 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:39.202444 2582 generic.go:356] "Generic (PLEG): container finished" podID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerID="008502ec7bcfd3423ff9029f3efdca9fef8f05e18632eb59bc9f8afa8a18fcdb" exitCode=0 Jul 16 17:35:39.202663 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:39.202532 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" event={"ID":"275d0233-fb85-4457-bbaf-5a6d5c8c39c4","Type":"ContainerDied","Data":"008502ec7bcfd3423ff9029f3efdca9fef8f05e18632eb59bc9f8afa8a18fcdb"} Jul 16 17:35:39.202663 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:39.202582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" event={"ID":"275d0233-fb85-4457-bbaf-5a6d5c8c39c4","Type":"ContainerStarted","Data":"879515fecf24337c977fba1afa141b1f3c2ab4fb8019b290fec258b916a68320"} Jul 16 17:35:43.218735 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:43.218695 2582 generic.go:356] "Generic (PLEG): container finished" podID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerID="9660af72b4ef2a601be8f6a690db6fe6a81ffbecb81495b799d483510341b05e" exitCode=0 Jul 16 17:35:43.219205 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:43.218786 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" event={"ID":"275d0233-fb85-4457-bbaf-5a6d5c8c39c4","Type":"ContainerDied","Data":"9660af72b4ef2a601be8f6a690db6fe6a81ffbecb81495b799d483510341b05e"} Jul 16 17:35:44.224545 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:44.224516 2582 generic.go:356] "Generic (PLEG): container finished" podID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerID="5f292c15698e3f23dd9afab4b1c87d8cd77e847f357bcb2ae73724ee58cf4278" exitCode=0 Jul 16 17:35:44.224921 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:44.224591 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" event={"ID":"275d0233-fb85-4457-bbaf-5a6d5c8c39c4","Type":"ContainerDied","Data":"5f292c15698e3f23dd9afab4b1c87d8cd77e847f357bcb2ae73724ee58cf4278"} Jul 16 17:35:45.350919 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.350891 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:45.379584 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.379554 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-bundle\") pod \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " Jul 16 17:35:45.379718 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.379597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2w52\" (UniqueName: \"kubernetes.io/projected/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-kube-api-access-v2w52\") pod \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " Jul 16 17:35:45.379718 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.379632 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-util\") pod \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\" (UID: \"275d0233-fb85-4457-bbaf-5a6d5c8c39c4\") " Jul 16 17:35:45.379986 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.379958 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-bundle" (OuterVolumeSpecName: "bundle") pod "275d0233-fb85-4457-bbaf-5a6d5c8c39c4" (UID: "275d0233-fb85-4457-bbaf-5a6d5c8c39c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:35:45.381749 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.381723 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-kube-api-access-v2w52" (OuterVolumeSpecName: "kube-api-access-v2w52") pod "275d0233-fb85-4457-bbaf-5a6d5c8c39c4" (UID: "275d0233-fb85-4457-bbaf-5a6d5c8c39c4"). InnerVolumeSpecName "kube-api-access-v2w52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:35:45.384371 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.384332 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-util" (OuterVolumeSpecName: "util") pod "275d0233-fb85-4457-bbaf-5a6d5c8c39c4" (UID: "275d0233-fb85-4457-bbaf-5a6d5c8c39c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:35:45.480392 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.480325 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-bundle\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:35:45.480392 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.480345 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2w52\" (UniqueName: \"kubernetes.io/projected/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-kube-api-access-v2w52\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:35:45.480392 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:45.480353 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/275d0233-fb85-4457-bbaf-5a6d5c8c39c4-util\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:35:46.232778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:46.232750 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" Jul 16 17:35:46.232942 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:46.232743 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fltkwn" event={"ID":"275d0233-fb85-4457-bbaf-5a6d5c8c39c4","Type":"ContainerDied","Data":"879515fecf24337c977fba1afa141b1f3c2ab4fb8019b290fec258b916a68320"} Jul 16 17:35:46.232942 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:46.232871 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="879515fecf24337c977fba1afa141b1f3c2ab4fb8019b290fec258b916a68320" Jul 16 17:35:49.203736 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.203701 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rlxkv"] Jul 16 17:35:49.208556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.204512 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerName="pull" Jul 16 17:35:49.208556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.204537 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerName="pull" Jul 16 17:35:49.208556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.204565 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerName="util" Jul 16 17:35:49.208556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.204574 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerName="util" Jul 16 17:35:49.208556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.204590 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerName="extract" Jul 16 17:35:49.208556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.204598 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerName="extract" Jul 16 17:35:49.208556 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.204723 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="275d0233-fb85-4457-bbaf-5a6d5c8c39c4" containerName="extract" Jul 16 17:35:49.209322 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.209304 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rlxkv" Jul 16 17:35:49.215531 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.215514 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 17:35:49.215742 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.215721 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zq5cr\"" Jul 16 17:35:49.216696 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.216681 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 17:35:49.221940 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.221919 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rlxkv"] Jul 16 17:35:49.310403 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.310369 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2z9k\" (UniqueName: \"kubernetes.io/projected/e817c71d-8d8c-4837-a629-0002bce7ac74-kube-api-access-j2z9k\") pod \"cert-manager-55b4d954d5-rlxkv\" (UID: \"e817c71d-8d8c-4837-a629-0002bce7ac74\") " pod="cert-manager/cert-manager-55b4d954d5-rlxkv" Jul 16 17:35:49.310565 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.310417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e817c71d-8d8c-4837-a629-0002bce7ac74-bound-sa-token\") pod \"cert-manager-55b4d954d5-rlxkv\" (UID: \"e817c71d-8d8c-4837-a629-0002bce7ac74\") " pod="cert-manager/cert-manager-55b4d954d5-rlxkv" Jul 16 17:35:49.411008 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.410975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2z9k\" (UniqueName: \"kubernetes.io/projected/e817c71d-8d8c-4837-a629-0002bce7ac74-kube-api-access-j2z9k\") pod \"cert-manager-55b4d954d5-rlxkv\" (UID: \"e817c71d-8d8c-4837-a629-0002bce7ac74\") " pod="cert-manager/cert-manager-55b4d954d5-rlxkv" Jul 16 17:35:49.411152 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.411025 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e817c71d-8d8c-4837-a629-0002bce7ac74-bound-sa-token\") pod \"cert-manager-55b4d954d5-rlxkv\" (UID: \"e817c71d-8d8c-4837-a629-0002bce7ac74\") " pod="cert-manager/cert-manager-55b4d954d5-rlxkv" Jul 16 17:35:49.449549 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.449522 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e817c71d-8d8c-4837-a629-0002bce7ac74-bound-sa-token\") pod \"cert-manager-55b4d954d5-rlxkv\" (UID: \"e817c71d-8d8c-4837-a629-0002bce7ac74\") " pod="cert-manager/cert-manager-55b4d954d5-rlxkv" Jul 16 17:35:49.451097 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.451081 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2z9k\" (UniqueName: \"kubernetes.io/projected/e817c71d-8d8c-4837-a629-0002bce7ac74-kube-api-access-j2z9k\") pod \"cert-manager-55b4d954d5-rlxkv\" (UID: \"e817c71d-8d8c-4837-a629-0002bce7ac74\") " pod="cert-manager/cert-manager-55b4d954d5-rlxkv" Jul 16 17:35:49.518370 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.518269 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rlxkv" Jul 16 17:35:49.674878 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:49.674848 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rlxkv"] Jul 16 17:35:49.678428 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:35:49.678394 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode817c71d_8d8c_4837_a629_0002bce7ac74.slice/crio-9e9e38fa6ef00d0cb5bc9cd6820dd3a389c6e210ab59333d4b08d5f5b10cc072 WatchSource:0}: Error finding container 9e9e38fa6ef00d0cb5bc9cd6820dd3a389c6e210ab59333d4b08d5f5b10cc072: Status 404 returned error can't find the container with id 9e9e38fa6ef00d0cb5bc9cd6820dd3a389c6e210ab59333d4b08d5f5b10cc072 Jul 16 17:35:50.249262 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:50.249228 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rlxkv" event={"ID":"e817c71d-8d8c-4837-a629-0002bce7ac74","Type":"ContainerStarted","Data":"9e9e38fa6ef00d0cb5bc9cd6820dd3a389c6e210ab59333d4b08d5f5b10cc072"} Jul 16 17:35:54.266568 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:54.266528 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rlxkv" event={"ID":"e817c71d-8d8c-4837-a629-0002bce7ac74","Type":"ContainerStarted","Data":"ac44bec4ea699263ec4b9a577690a09999b4a07afbc0d5f7594877b88ef35b7e"} Jul 16 17:35:54.288901 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:54.288838 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rlxkv" podStartSLOduration=1.7474259650000001 podStartE2EDuration="5.288820115s" totalImagesPullingTime="3.54139415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:54.288404889 +0000 UTC m=+416.040044447" watchObservedRunningTime="2026-07-16 17:35:54.288820115 +0000 UTC m=+416.040459673" Jul 16 17:35:56.156485 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.156454 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s"] Jul 16 17:35:56.159427 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.159411 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.162771 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.162732 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 17:35:56.162916 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.162818 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 17:35:56.163001 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.162988 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mzvtg\"" Jul 16 17:35:56.178576 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.178543 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s"] Jul 16 17:35:56.265822 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.265781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gqn\" (UniqueName: \"kubernetes.io/projected/1b8c7cef-c9de-4e33-9575-5944d1979a56-kube-api-access-l9gqn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.265822 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.265822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.266045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.265844 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.366552 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.366513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gqn\" (UniqueName: \"kubernetes.io/projected/1b8c7cef-c9de-4e33-9575-5944d1979a56-kube-api-access-l9gqn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.366717 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.366560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.366775 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.366753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.366948 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.366930 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.367072 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.367053 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.382992 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.382954 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gqn\" (UniqueName: \"kubernetes.io/projected/1b8c7cef-c9de-4e33-9575-5944d1979a56-kube-api-access-l9gqn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.468229 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.468145 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:35:56.607159 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:56.607136 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s"] Jul 16 17:35:56.609507 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:35:56.609478 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8c7cef_c9de_4e33_9575_5944d1979a56.slice/crio-d3c5b2bbe328bf59d7f87c7f81137a435e36c6553392f9811574e1d965262423 WatchSource:0}: Error finding container d3c5b2bbe328bf59d7f87c7f81137a435e36c6553392f9811574e1d965262423: Status 404 returned error can't find the container with id d3c5b2bbe328bf59d7f87c7f81137a435e36c6553392f9811574e1d965262423 Jul 16 17:35:57.279270 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:57.279232 2582 generic.go:356] "Generic (PLEG): container finished" podID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerID="6a8c6cdeec375a470717aa3c1ee9f3742335d3f01b37fdc22ca07d44278cde35" exitCode=0 Jul 16 17:35:57.279655 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:57.279324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" event={"ID":"1b8c7cef-c9de-4e33-9575-5944d1979a56","Type":"ContainerDied","Data":"6a8c6cdeec375a470717aa3c1ee9f3742335d3f01b37fdc22ca07d44278cde35"} Jul 16 17:35:57.279655 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:57.279381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" event={"ID":"1b8c7cef-c9de-4e33-9575-5944d1979a56","Type":"ContainerStarted","Data":"d3c5b2bbe328bf59d7f87c7f81137a435e36c6553392f9811574e1d965262423"} Jul 16 17:35:58.284947 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:58.284914 2582 generic.go:356] "Generic (PLEG): container finished" podID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerID="dc7cc768cfed757919880f6d9a602f0d2db4d1540d6b14050707c4dc4aaacd73" exitCode=0 Jul 16 17:35:58.285354 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:58.284961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" event={"ID":"1b8c7cef-c9de-4e33-9575-5944d1979a56","Type":"ContainerDied","Data":"dc7cc768cfed757919880f6d9a602f0d2db4d1540d6b14050707c4dc4aaacd73"} Jul 16 17:35:59.290707 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:59.290671 2582 generic.go:356] "Generic (PLEG): container finished" podID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerID="6a99d47ce7017a439cd6b7e03c184c6680164bcea7ce2936ea8041b54134fbc0" exitCode=0 Jul 16 17:35:59.290707 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:35:59.290711 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" event={"ID":"1b8c7cef-c9de-4e33-9575-5944d1979a56","Type":"ContainerDied","Data":"6a99d47ce7017a439cd6b7e03c184c6680164bcea7ce2936ea8041b54134fbc0"} Jul 16 17:36:00.441373 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.441344 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:36:00.494359 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.494316 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-bundle\") pod \"1b8c7cef-c9de-4e33-9575-5944d1979a56\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " Jul 16 17:36:00.494359 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.494373 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-util\") pod \"1b8c7cef-c9de-4e33-9575-5944d1979a56\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " Jul 16 17:36:00.494621 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.494423 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9gqn\" (UniqueName: \"kubernetes.io/projected/1b8c7cef-c9de-4e33-9575-5944d1979a56-kube-api-access-l9gqn\") pod \"1b8c7cef-c9de-4e33-9575-5944d1979a56\" (UID: \"1b8c7cef-c9de-4e33-9575-5944d1979a56\") " Jul 16 17:36:00.495148 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.495090 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-bundle" (OuterVolumeSpecName: "bundle") pod "1b8c7cef-c9de-4e33-9575-5944d1979a56" (UID: "1b8c7cef-c9de-4e33-9575-5944d1979a56"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:36:00.496680 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.496652 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8c7cef-c9de-4e33-9575-5944d1979a56-kube-api-access-l9gqn" (OuterVolumeSpecName: "kube-api-access-l9gqn") pod "1b8c7cef-c9de-4e33-9575-5944d1979a56" (UID: "1b8c7cef-c9de-4e33-9575-5944d1979a56"). InnerVolumeSpecName "kube-api-access-l9gqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:00.499518 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.499495 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-util" (OuterVolumeSpecName: "util") pod "1b8c7cef-c9de-4e33-9575-5944d1979a56" (UID: "1b8c7cef-c9de-4e33-9575-5944d1979a56"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:36:00.596050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.595959 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-bundle\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:00.596050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.595990 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b8c7cef-c9de-4e33-9575-5944d1979a56-util\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:00.596050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:00.595999 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9gqn\" (UniqueName: \"kubernetes.io/projected/1b8c7cef-c9de-4e33-9575-5944d1979a56-kube-api-access-l9gqn\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:01.299954 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:01.299917 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" event={"ID":"1b8c7cef-c9de-4e33-9575-5944d1979a56","Type":"ContainerDied","Data":"d3c5b2bbe328bf59d7f87c7f81137a435e36c6553392f9811574e1d965262423"} Jul 16 17:36:01.300118 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:01.299965 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afths4s" Jul 16 17:36:01.300118 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:01.299973 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3c5b2bbe328bf59d7f87c7f81137a435e36c6553392f9811574e1d965262423" Jul 16 17:36:09.591860 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.591823 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m"] Jul 16 17:36:09.592259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.592104 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerName="extract" Jul 16 17:36:09.592259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.592115 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerName="extract" Jul 16 17:36:09.592259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.592135 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerName="util" Jul 16 17:36:09.592259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.592140 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerName="util" Jul 16 17:36:09.592259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.592146 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerName="pull" Jul 16 17:36:09.592259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.592151 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerName="pull" Jul 16 17:36:09.592259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.592194 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8c7cef-c9de-4e33-9575-5944d1979a56" containerName="extract" Jul 16 17:36:09.595363 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.595346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.600486 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.600460 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 17:36:09.600900 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.600879 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 17:36:09.601036 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.601019 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 17:36:09.601162 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.601142 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gfh6s\"" Jul 16 17:36:09.601249 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.601191 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 17:36:09.628468 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.628439 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m"] Jul 16 17:36:09.667084 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.667052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8s7n\" (UniqueName: \"kubernetes.io/projected/07dce6dc-1128-4309-96c4-984089699d19-kube-api-access-v8s7n\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.667242 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.667114 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dce6dc-1128-4309-96c4-984089699d19-tmp\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.667242 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.667161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07dce6dc-1128-4309-96c4-984089699d19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.667242 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.667200 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07dce6dc-1128-4309-96c4-984089699d19-manifests\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.667242 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.667236 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07dce6dc-1128-4309-96c4-984089699d19-webhook-cert\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.768443 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.768387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8s7n\" (UniqueName: \"kubernetes.io/projected/07dce6dc-1128-4309-96c4-984089699d19-kube-api-access-v8s7n\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.768654 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.768472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dce6dc-1128-4309-96c4-984089699d19-tmp\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.768654 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.768501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07dce6dc-1128-4309-96c4-984089699d19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.768654 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.768534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07dce6dc-1128-4309-96c4-984089699d19-manifests\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.768654 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.768582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07dce6dc-1128-4309-96c4-984089699d19-webhook-cert\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.768954 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.768930 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07dce6dc-1128-4309-96c4-984089699d19-manifests\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.770939 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.770907 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dce6dc-1128-4309-96c4-984089699d19-tmp\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.771144 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.771125 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07dce6dc-1128-4309-96c4-984089699d19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.771206 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.771147 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07dce6dc-1128-4309-96c4-984089699d19-webhook-cert\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.781845 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.781808 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8s7n\" (UniqueName: \"kubernetes.io/projected/07dce6dc-1128-4309-96c4-984089699d19-kube-api-access-v8s7n\") pod \"opendatahub-operator-controller-manager-586f4b4fcb-cd69m\" (UID: \"07dce6dc-1128-4309-96c4-984089699d19\") " pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:09.906133 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:09.906043 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:10.051633 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:10.051602 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m"] Jul 16 17:36:10.054693 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:36:10.054667 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07dce6dc_1128_4309_96c4_984089699d19.slice/crio-f832191f06c1639c9716fb75a60ba48a89153a761488ee9a9a2afe8c6effcbe7 WatchSource:0}: Error finding container f832191f06c1639c9716fb75a60ba48a89153a761488ee9a9a2afe8c6effcbe7: Status 404 returned error can't find the container with id f832191f06c1639c9716fb75a60ba48a89153a761488ee9a9a2afe8c6effcbe7 Jul 16 17:36:10.335061 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:10.335017 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" event={"ID":"07dce6dc-1128-4309-96c4-984089699d19","Type":"ContainerStarted","Data":"f832191f06c1639c9716fb75a60ba48a89153a761488ee9a9a2afe8c6effcbe7"} Jul 16 17:36:13.349996 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:13.349416 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" event={"ID":"07dce6dc-1128-4309-96c4-984089699d19","Type":"ContainerStarted","Data":"f0f5f1eb482810a6b213a4bee16ae91f6e8a0dcaf976a77060201f4cdf9cee1a"} Jul 16 17:36:14.355148 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:14.355115 2582 generic.go:356] "Generic (PLEG): container finished" podID="07dce6dc-1128-4309-96c4-984089699d19" containerID="f0f5f1eb482810a6b213a4bee16ae91f6e8a0dcaf976a77060201f4cdf9cee1a" exitCode=0 Jul 16 17:36:14.355602 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:14.355205 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" event={"ID":"07dce6dc-1128-4309-96c4-984089699d19","Type":"ContainerDied","Data":"f0f5f1eb482810a6b213a4bee16ae91f6e8a0dcaf976a77060201f4cdf9cee1a"} Jul 16 17:36:16.365170 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:16.365132 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" event={"ID":"07dce6dc-1128-4309-96c4-984089699d19","Type":"ContainerStarted","Data":"5d9b5a4d537a415f5f4e3a08a3eade190c5d7647de86dac73fbe351188611ec7"} Jul 16 17:36:16.365624 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:16.365273 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:16.401389 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:16.401340 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" podStartSLOduration=2.303465864 podStartE2EDuration="7.401322614s" totalImagesPullingTime="4.077932326s" totalInitContainerRuntime="1.019924424s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:16.400136623 +0000 UTC m=+438.151776182" watchObservedRunningTime="2026-07-16 17:36:16.401322614 +0000 UTC m=+438.152962172" Jul 16 17:36:27.371702 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:27.371626 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-586f4b4fcb-cd69m" Jul 16 17:36:29.605150 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.605110 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl"] Jul 16 17:36:29.614104 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.614079 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.617224 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.617206 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 17:36:29.617911 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.617891 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 17:36:29.621637 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.621620 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pjrmn\"" Jul 16 17:36:29.625961 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.625940 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl"] Jul 16 17:36:29.732634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.732594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45b2bca5-c938-45fd-b8e3-b010d365d933-tmp\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.732825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.732654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d598m\" (UniqueName: \"kubernetes.io/projected/45b2bca5-c938-45fd-b8e3-b010d365d933-kube-api-access-d598m\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.732825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.732718 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b2bca5-c938-45fd-b8e3-b010d365d933-tls-certs\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.818438 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.818405 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t6d4r"] Jul 16 17:36:29.821561 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.821543 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:29.826192 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.826173 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 16 17:36:29.826309 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.826292 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b8nkk\"" Jul 16 17:36:29.833709 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.833685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d598m\" (UniqueName: \"kubernetes.io/projected/45b2bca5-c938-45fd-b8e3-b010d365d933-kube-api-access-d598m\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.833804 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.833725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b2bca5-c938-45fd-b8e3-b010d365d933-tls-certs\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.833804 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.833770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45b2bca5-c938-45fd-b8e3-b010d365d933-tmp\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.836510 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.836474 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45b2bca5-c938-45fd-b8e3-b010d365d933-tmp\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.836643 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.836606 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b2bca5-c938-45fd-b8e3-b010d365d933-tls-certs\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.837651 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.837631 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t6d4r"] Jul 16 17:36:29.849874 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.849848 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d598m\" (UniqueName: \"kubernetes.io/projected/45b2bca5-c938-45fd-b8e3-b010d365d933-kube-api-access-d598m\") pod \"kube-auth-proxy-7f8cccc594-hpkrl\" (UID: \"45b2bca5-c938-45fd-b8e3-b010d365d933\") " pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.923706 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.923615 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" Jul 16 17:36:29.935134 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.935108 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f77167-bd56-464f-b6e8-c41e91b71181-cert\") pod \"odh-model-controller-7998c67dc4-t6d4r\" (UID: \"e9f77167-bd56-464f-b6e8-c41e91b71181\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:29.935199 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:29.935145 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2p9\" (UniqueName: \"kubernetes.io/projected/e9f77167-bd56-464f-b6e8-c41e91b71181-kube-api-access-hv2p9\") pod \"odh-model-controller-7998c67dc4-t6d4r\" (UID: \"e9f77167-bd56-464f-b6e8-c41e91b71181\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:30.035944 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.035907 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f77167-bd56-464f-b6e8-c41e91b71181-cert\") pod \"odh-model-controller-7998c67dc4-t6d4r\" (UID: \"e9f77167-bd56-464f-b6e8-c41e91b71181\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:30.035944 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.035946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2p9\" (UniqueName: \"kubernetes.io/projected/e9f77167-bd56-464f-b6e8-c41e91b71181-kube-api-access-hv2p9\") pod \"odh-model-controller-7998c67dc4-t6d4r\" (UID: \"e9f77167-bd56-464f-b6e8-c41e91b71181\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:30.038774 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.038736 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f77167-bd56-464f-b6e8-c41e91b71181-cert\") pod \"odh-model-controller-7998c67dc4-t6d4r\" (UID: \"e9f77167-bd56-464f-b6e8-c41e91b71181\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:30.073866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.073835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2p9\" (UniqueName: \"kubernetes.io/projected/e9f77167-bd56-464f-b6e8-c41e91b71181-kube-api-access-hv2p9\") pod \"odh-model-controller-7998c67dc4-t6d4r\" (UID: \"e9f77167-bd56-464f-b6e8-c41e91b71181\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:30.101743 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.101703 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl"] Jul 16 17:36:30.104934 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:36:30.104909 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b2bca5_c938_45fd_b8e3_b010d365d933.slice/crio-374a23421b2c8e8d34a2b4a55e4fbcad0b682df9ca21cdb2869fb9959907ec30 WatchSource:0}: Error finding container 374a23421b2c8e8d34a2b4a55e4fbcad0b682df9ca21cdb2869fb9959907ec30: Status 404 returned error can't find the container with id 374a23421b2c8e8d34a2b4a55e4fbcad0b682df9ca21cdb2869fb9959907ec30 Jul 16 17:36:30.130967 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.130927 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:30.270864 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.270833 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t6d4r"] Jul 16 17:36:30.274423 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:36:30.274400 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f77167_bd56_464f_b6e8_c41e91b71181.slice/crio-ff2c9763aa33d6ccb73f9781b8294c06b1af71814c7201672aa88034a579df2d WatchSource:0}: Error finding container ff2c9763aa33d6ccb73f9781b8294c06b1af71814c7201672aa88034a579df2d: Status 404 returned error can't find the container with id ff2c9763aa33d6ccb73f9781b8294c06b1af71814c7201672aa88034a579df2d Jul 16 17:36:30.421250 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.421208 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" event={"ID":"45b2bca5-c938-45fd-b8e3-b010d365d933","Type":"ContainerStarted","Data":"374a23421b2c8e8d34a2b4a55e4fbcad0b682df9ca21cdb2869fb9959907ec30"} Jul 16 17:36:30.422567 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.422537 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" event={"ID":"e9f77167-bd56-464f-b6e8-c41e91b71181","Type":"ContainerStarted","Data":"ff2c9763aa33d6ccb73f9781b8294c06b1af71814c7201672aa88034a579df2d"} Jul 16 17:36:30.619926 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.619889 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r59fz"] Jul 16 17:36:30.624174 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.624155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:30.627245 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.627215 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 16 17:36:30.635739 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.635714 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r59fz"] Jul 16 17:36:30.742725 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.742688 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecc13a7-925c-4045-9da0-51909d1cf391-tls-certs\") pod \"model-serving-api-54d89bd5fc-r59fz\" (UID: \"7ecc13a7-925c-4045-9da0-51909d1cf391\") " pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:30.742925 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.742781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5j9p\" (UniqueName: \"kubernetes.io/projected/7ecc13a7-925c-4045-9da0-51909d1cf391-kube-api-access-x5j9p\") pod \"model-serving-api-54d89bd5fc-r59fz\" (UID: \"7ecc13a7-925c-4045-9da0-51909d1cf391\") " pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:30.844136 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.844091 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecc13a7-925c-4045-9da0-51909d1cf391-tls-certs\") pod \"model-serving-api-54d89bd5fc-r59fz\" (UID: \"7ecc13a7-925c-4045-9da0-51909d1cf391\") " pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:30.844363 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.844200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5j9p\" (UniqueName: \"kubernetes.io/projected/7ecc13a7-925c-4045-9da0-51909d1cf391-kube-api-access-x5j9p\") pod \"model-serving-api-54d89bd5fc-r59fz\" (UID: \"7ecc13a7-925c-4045-9da0-51909d1cf391\") " pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:30.847731 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.847709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecc13a7-925c-4045-9da0-51909d1cf391-tls-certs\") pod \"model-serving-api-54d89bd5fc-r59fz\" (UID: \"7ecc13a7-925c-4045-9da0-51909d1cf391\") " pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:30.863536 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.863505 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5j9p\" (UniqueName: \"kubernetes.io/projected/7ecc13a7-925c-4045-9da0-51909d1cf391-kube-api-access-x5j9p\") pod \"model-serving-api-54d89bd5fc-r59fz\" (UID: \"7ecc13a7-925c-4045-9da0-51909d1cf391\") " pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:30.935384 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:30.935260 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:31.188152 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.188123 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r59fz"] Jul 16 17:36:31.190898 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:36:31.190871 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc13a7_925c_4045_9da0_51909d1cf391.slice/crio-87f756dea9682ebc97e06b044d3269a495a8eae2f60ee553ff7e42bb2030beaa WatchSource:0}: Error finding container 87f756dea9682ebc97e06b044d3269a495a8eae2f60ee553ff7e42bb2030beaa: Status 404 returned error can't find the container with id 87f756dea9682ebc97e06b044d3269a495a8eae2f60ee553ff7e42bb2030beaa Jul 16 17:36:31.428002 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.427963 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" event={"ID":"7ecc13a7-925c-4045-9da0-51909d1cf391","Type":"ContainerStarted","Data":"87f756dea9682ebc97e06b044d3269a495a8eae2f60ee553ff7e42bb2030beaa"} Jul 16 17:36:31.855022 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.854989 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-m28kv"] Jul 16 17:36:31.858425 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.858400 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:31.862753 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.862539 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l4zzw\"" Jul 16 17:36:31.862753 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.862582 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 16 17:36:31.862753 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.862650 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 17:36:31.862998 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.862843 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 16 17:36:31.863170 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.863153 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 16 17:36:31.879658 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.879587 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-m28kv"] Jul 16 17:36:31.952922 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.952356 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5332e69-6991-44c8-811f-0786619b8fd5-local-certs\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:31.952922 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.952405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-istio-token\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:31.952922 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.952438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:31.952922 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.952492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-cacerts\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:31.952922 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.952713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:31.952922 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.952796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:31.952922 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:31.952831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjb8\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-kube-api-access-lvjb8\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.054054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.054004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5332e69-6991-44c8-811f-0786619b8fd5-local-certs\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.054054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.054056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-istio-token\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.054458 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.054083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.054458 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.054112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-cacerts\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.054458 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.054166 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.054458 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.054238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.054458 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.054271 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjb8\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-kube-api-access-lvjb8\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.055433 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.055167 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.058314 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.058259 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.059597 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.059507 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5332e69-6991-44c8-811f-0786619b8fd5-local-certs\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.061174 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.061137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-cacerts\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.061506 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.061480 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.070766 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.070709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjb8\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-kube-api-access-lvjb8\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.074763 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.074733 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-istio-token\") pod \"istiod-openshift-gateway-7574459656-m28kv\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.171553 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.171466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:32.366848 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:32.366772 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-m28kv"] Jul 16 17:36:33.965720 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:36:33.965685 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5332e69_6991_44c8_811f_0786619b8fd5.slice/crio-a4559e68791199176158a9465da30ab08e2c9e8e23eff95643621a09c107a2e5 WatchSource:0}: Error finding container a4559e68791199176158a9465da30ab08e2c9e8e23eff95643621a09c107a2e5: Status 404 returned error can't find the container with id a4559e68791199176158a9465da30ab08e2c9e8e23eff95643621a09c107a2e5 Jul 16 17:36:34.444552 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:34.444513 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" event={"ID":"a5332e69-6991-44c8-811f-0786619b8fd5","Type":"ContainerStarted","Data":"a4559e68791199176158a9465da30ab08e2c9e8e23eff95643621a09c107a2e5"} Jul 16 17:36:35.450570 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:35.450532 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" event={"ID":"45b2bca5-c938-45fd-b8e3-b010d365d933","Type":"ContainerStarted","Data":"a35242cc5f71b109b8937b51b6c8a5461580cac558a182e3e5ff2b1511be8fcf"} Jul 16 17:36:35.452071 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:35.452045 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" event={"ID":"e9f77167-bd56-464f-b6e8-c41e91b71181","Type":"ContainerStarted","Data":"13b566bc2823a150c80977820a1503e63afb1d4e1dbf1eaa8e574030bf5e8364"} Jul 16 17:36:35.452207 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:35.452130 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:35.453410 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:35.453392 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" event={"ID":"7ecc13a7-925c-4045-9da0-51909d1cf391","Type":"ContainerStarted","Data":"dab3b887d2d92f84fe7f10b02679713653f22b266653e3daef84656debbc06c0"} Jul 16 17:36:35.453525 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:35.453514 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:35.470609 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:35.470565 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f8cccc594-hpkrl" podStartSLOduration=1.9991904809999999 podStartE2EDuration="6.470546733s" totalImagesPullingTime="4.471356252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:35.469567802 +0000 UTC m=+457.221207360" watchObservedRunningTime="2026-07-16 17:36:35.470546733 +0000 UTC m=+457.222186290" Jul 16 17:36:35.490325 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:35.490232 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" podStartSLOduration=2.181116737 podStartE2EDuration="6.490208208s" totalImagesPullingTime="4.309091471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:35.488571432 +0000 UTC m=+457.240211022" watchObservedRunningTime="2026-07-16 17:36:35.490208208 +0000 UTC m=+457.241847767" Jul 16 17:36:35.513326 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:35.513256 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" podStartSLOduration=2.132199399 podStartE2EDuration="5.51323946s" totalImagesPullingTime="3.381040061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:35.512205052 +0000 UTC m=+457.263844629" watchObservedRunningTime="2026-07-16 17:36:35.51323946 +0000 UTC m=+457.264879019" Jul 16 17:36:37.755406 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:37.755369 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:36:37.755671 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:37.755435 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:36:38.470574 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:38.470540 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" event={"ID":"a5332e69-6991-44c8-811f-0786619b8fd5","Type":"ContainerStarted","Data":"5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c"} Jul 16 17:36:38.470761 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:38.470693 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:38.472741 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:38.472714 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-m28kv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 17:36:38.472876 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:38.472772 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" podUID="a5332e69-6991-44c8-811f-0786619b8fd5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:36:38.530555 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:38.530507 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" podStartSLOduration=3.743292424 podStartE2EDuration="7.53049113s" totalImagesPullingTime="3.787198706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:38.529464649 +0000 UTC m=+460.281104204" watchObservedRunningTime="2026-07-16 17:36:38.53049113 +0000 UTC m=+460.282130688" Jul 16 17:36:39.475327 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.475274 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-m28kv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 17:36:39.475719 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.475350 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" podUID="a5332e69-6991-44c8-811f-0786619b8fd5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:36:39.612408 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.612371 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd"] Jul 16 17:36:39.615767 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.615744 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.619948 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.619929 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nmrgr\"" Jul 16 17:36:39.639376 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.639349 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd"] Jul 16 17:36:39.721585 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc90a598-4135-43a9-a06e-db32379f92a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.721773 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721591 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2vdg\" (UniqueName: \"kubernetes.io/projected/cc90a598-4135-43a9-a06e-db32379f92a4-kube-api-access-s2vdg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.721773 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721661 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.721773 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.721773 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721710 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc90a598-4135-43a9-a06e-db32379f92a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.721773 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721754 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.721986 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.721986 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.721986 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.721846 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc90a598-4135-43a9-a06e-db32379f92a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.822902 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.822864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.822902 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.822900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823152 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.822919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc90a598-4135-43a9-a06e-db32379f92a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823152 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823060 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823152 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823324 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823324 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc90a598-4135-43a9-a06e-db32379f92a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823324 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc90a598-4135-43a9-a06e-db32379f92a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823473 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2vdg\" (UniqueName: \"kubernetes.io/projected/cc90a598-4135-43a9-a06e-db32379f92a4-kube-api-access-s2vdg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823642 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823585 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823731 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823645 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823793 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823759 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823882 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.823990 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.823974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc90a598-4135-43a9-a06e-db32379f92a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.825428 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.825406 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc90a598-4135-43a9-a06e-db32379f92a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.826052 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.826032 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc90a598-4135-43a9-a06e-db32379f92a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.838261 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.838223 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2vdg\" (UniqueName: \"kubernetes.io/projected/cc90a598-4135-43a9-a06e-db32379f92a4-kube-api-access-s2vdg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.838875 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.838856 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc90a598-4135-43a9-a06e-db32379f92a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46c8gd\" (UID: \"cc90a598-4135-43a9-a06e-db32379f92a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:39.925356 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:39.925309 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:40.114174 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.113975 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd"] Jul 16 17:36:40.116668 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:36:40.116633 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc90a598_4135_43a9_a06e_db32379f92a4.slice/crio-1a2385ab38a1285430d8cc03cce84b94544b668c14e410ddbdcc4cc942283dab WatchSource:0}: Error finding container 1a2385ab38a1285430d8cc03cce84b94544b668c14e410ddbdcc4cc942283dab: Status 404 returned error can't find the container with id 1a2385ab38a1285430d8cc03cce84b94544b668c14e410ddbdcc4cc942283dab Jul 16 17:36:40.479567 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.479484 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" event={"ID":"cc90a598-4135-43a9-a06e-db32379f92a4","Type":"ContainerStarted","Data":"1a2385ab38a1285430d8cc03cce84b94544b668c14e410ddbdcc4cc942283dab"} Jul 16 17:36:40.879545 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.879511 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd"] Jul 16 17:36:40.886946 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.886915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:40.893537 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.893512 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 16 17:36:40.893862 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.893844 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8kcjw\"" Jul 16 17:36:40.895337 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.895315 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f"] Jul 16 17:36:40.898385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.898362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:40.904812 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.904796 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 16 17:36:40.904921 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.904839 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-nbbsv\"" Jul 16 17:36:40.919146 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.919117 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd"] Jul 16 17:36:40.986177 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:40.986139 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f"] Jul 16 17:36:41.033596 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.033559 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43fae792-e9bd-4de3-bd7c-8a327a7e40de-cert\") pod \"kserve-controller-manager-d6f4b65b8-6wn7f\" (UID: \"43fae792-e9bd-4de3-bd7c-8a327a7e40de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:41.033733 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.033617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799t7\" (UniqueName: \"kubernetes.io/projected/7861aee2-fad9-4cc1-a518-268115eb3265-kube-api-access-799t7\") pod \"llmisvc-controller-manager-777c6bd7d5-pp4dd\" (UID: \"7861aee2-fad9-4cc1-a518-268115eb3265\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:41.033733 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.033672 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4hw\" (UniqueName: \"kubernetes.io/projected/43fae792-e9bd-4de3-bd7c-8a327a7e40de-kube-api-access-gk4hw\") pod \"kserve-controller-manager-d6f4b65b8-6wn7f\" (UID: \"43fae792-e9bd-4de3-bd7c-8a327a7e40de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:41.033733 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.033702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7861aee2-fad9-4cc1-a518-268115eb3265-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pp4dd\" (UID: \"7861aee2-fad9-4cc1-a518-268115eb3265\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:41.134712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.134622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4hw\" (UniqueName: \"kubernetes.io/projected/43fae792-e9bd-4de3-bd7c-8a327a7e40de-kube-api-access-gk4hw\") pod \"kserve-controller-manager-d6f4b65b8-6wn7f\" (UID: \"43fae792-e9bd-4de3-bd7c-8a327a7e40de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:41.134712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.134664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7861aee2-fad9-4cc1-a518-268115eb3265-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pp4dd\" (UID: \"7861aee2-fad9-4cc1-a518-268115eb3265\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:41.134712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.134712 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43fae792-e9bd-4de3-bd7c-8a327a7e40de-cert\") pod \"kserve-controller-manager-d6f4b65b8-6wn7f\" (UID: \"43fae792-e9bd-4de3-bd7c-8a327a7e40de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:41.134920 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.134744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799t7\" (UniqueName: \"kubernetes.io/projected/7861aee2-fad9-4cc1-a518-268115eb3265-kube-api-access-799t7\") pod \"llmisvc-controller-manager-777c6bd7d5-pp4dd\" (UID: \"7861aee2-fad9-4cc1-a518-268115eb3265\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:41.137345 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.137320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43fae792-e9bd-4de3-bd7c-8a327a7e40de-cert\") pod \"kserve-controller-manager-d6f4b65b8-6wn7f\" (UID: \"43fae792-e9bd-4de3-bd7c-8a327a7e40de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:41.137471 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.137410 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7861aee2-fad9-4cc1-a518-268115eb3265-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pp4dd\" (UID: \"7861aee2-fad9-4cc1-a518-268115eb3265\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:41.145697 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.145672 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4hw\" (UniqueName: \"kubernetes.io/projected/43fae792-e9bd-4de3-bd7c-8a327a7e40de-kube-api-access-gk4hw\") pod \"kserve-controller-manager-d6f4b65b8-6wn7f\" (UID: \"43fae792-e9bd-4de3-bd7c-8a327a7e40de\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:41.147210 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.147188 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-799t7\" (UniqueName: \"kubernetes.io/projected/7861aee2-fad9-4cc1-a518-268115eb3265-kube-api-access-799t7\") pod \"llmisvc-controller-manager-777c6bd7d5-pp4dd\" (UID: \"7861aee2-fad9-4cc1-a518-268115eb3265\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:41.198063 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.198019 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:41.207375 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.207348 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:41.360245 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:36:41.360212 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7861aee2_fad9_4cc1_a518_268115eb3265.slice/crio-d7d00ad422f4dc5657abc0b952d2c580a475770b9643dc93ad5e767915e2fb47 WatchSource:0}: Error finding container d7d00ad422f4dc5657abc0b952d2c580a475770b9643dc93ad5e767915e2fb47: Status 404 returned error can't find the container with id d7d00ad422f4dc5657abc0b952d2c580a475770b9643dc93ad5e767915e2fb47 Jul 16 17:36:41.363658 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.363635 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd"] Jul 16 17:36:41.387821 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.387786 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f"] Jul 16 17:36:41.390236 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:36:41.390210 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fae792_e9bd_4de3_bd7c_8a327a7e40de.slice/crio-615e66c3bdcbfaf45cec1aa52357fa3f389cb34cdafc35f1890841f5bf53214c WatchSource:0}: Error finding container 615e66c3bdcbfaf45cec1aa52357fa3f389cb34cdafc35f1890841f5bf53214c: Status 404 returned error can't find the container with id 615e66c3bdcbfaf45cec1aa52357fa3f389cb34cdafc35f1890841f5bf53214c Jul 16 17:36:41.485775 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.485731 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" event={"ID":"7861aee2-fad9-4cc1-a518-268115eb3265","Type":"ContainerStarted","Data":"d7d00ad422f4dc5657abc0b952d2c580a475770b9643dc93ad5e767915e2fb47"} Jul 16 17:36:41.487097 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:41.487069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" event={"ID":"43fae792-e9bd-4de3-bd7c-8a327a7e40de","Type":"ContainerStarted","Data":"615e66c3bdcbfaf45cec1aa52357fa3f389cb34cdafc35f1890841f5bf53214c"} Jul 16 17:36:42.475931 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:42.475899 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:42.951425 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:42.951388 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-m28kv"] Jul 16 17:36:42.951910 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:42.951581 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" podUID="a5332e69-6991-44c8-811f-0786619b8fd5" containerName="discovery" containerID="cri-o://5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c" gracePeriod=30 Jul 16 17:36:44.105674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.105652 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:44.123302 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.123258 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:36:44.123400 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.123345 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:36:44.123400 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.123377 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:36:44.167214 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.167191 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5332e69-6991-44c8-811f-0786619b8fd5-local-certs\") pod \"a5332e69-6991-44c8-811f-0786619b8fd5\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " Jul 16 17:36:44.167370 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.167220 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-istio-token\") pod \"a5332e69-6991-44c8-811f-0786619b8fd5\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " Jul 16 17:36:44.167370 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.167241 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-kubeconfig\") pod \"a5332e69-6991-44c8-811f-0786619b8fd5\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " Jul 16 17:36:44.167370 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.167274 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-ca-configmap\") pod \"a5332e69-6991-44c8-811f-0786619b8fd5\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " Jul 16 17:36:44.167533 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.167400 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvjb8\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-kube-api-access-lvjb8\") pod \"a5332e69-6991-44c8-811f-0786619b8fd5\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " Jul 16 17:36:44.167533 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.167437 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-cacerts\") pod \"a5332e69-6991-44c8-811f-0786619b8fd5\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " Jul 16 17:36:44.167533 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.167475 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-dns-cert\") pod \"a5332e69-6991-44c8-811f-0786619b8fd5\" (UID: \"a5332e69-6991-44c8-811f-0786619b8fd5\") " Jul 16 17:36:44.167729 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.167689 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a5332e69-6991-44c8-811f-0786619b8fd5" (UID: "a5332e69-6991-44c8-811f-0786619b8fd5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:36:44.169727 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.169691 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-kube-api-access-lvjb8" (OuterVolumeSpecName: "kube-api-access-lvjb8") pod "a5332e69-6991-44c8-811f-0786619b8fd5" (UID: "a5332e69-6991-44c8-811f-0786619b8fd5"). InnerVolumeSpecName "kube-api-access-lvjb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:44.169807 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.169729 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a5332e69-6991-44c8-811f-0786619b8fd5" (UID: "a5332e69-6991-44c8-811f-0786619b8fd5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:44.169989 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.169970 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5332e69-6991-44c8-811f-0786619b8fd5-local-certs" (OuterVolumeSpecName: "local-certs") pod "a5332e69-6991-44c8-811f-0786619b8fd5" (UID: "a5332e69-6991-44c8-811f-0786619b8fd5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:36:44.170042 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.169981 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5332e69-6991-44c8-811f-0786619b8fd5" (UID: "a5332e69-6991-44c8-811f-0786619b8fd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:44.170162 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.170145 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a5332e69-6991-44c8-811f-0786619b8fd5" (UID: "a5332e69-6991-44c8-811f-0786619b8fd5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:44.170201 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.170161 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-cacerts" (OuterVolumeSpecName: "cacerts") pod "a5332e69-6991-44c8-811f-0786619b8fd5" (UID: "a5332e69-6991-44c8-811f-0786619b8fd5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:44.268939 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.268904 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-ca-configmap\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:44.268939 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.268935 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvjb8\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-kube-api-access-lvjb8\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:44.268939 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.268948 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-cacerts\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:44.269166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.268960 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-csr-dns-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:44.269166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.268971 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5332e69-6991-44c8-811f-0786619b8fd5-local-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:44.269166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.268982 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5332e69-6991-44c8-811f-0786619b8fd5-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:44.269166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.268991 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a5332e69-6991-44c8-811f-0786619b8fd5-istio-kubeconfig\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:36:44.504485 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.504449 2582 generic.go:356] "Generic (PLEG): container finished" podID="a5332e69-6991-44c8-811f-0786619b8fd5" containerID="5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c" exitCode=0 Jul 16 17:36:44.504653 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.504497 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" Jul 16 17:36:44.504653 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.504525 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" event={"ID":"a5332e69-6991-44c8-811f-0786619b8fd5","Type":"ContainerDied","Data":"5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c"} Jul 16 17:36:44.504653 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.504572 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-m28kv" event={"ID":"a5332e69-6991-44c8-811f-0786619b8fd5","Type":"ContainerDied","Data":"a4559e68791199176158a9465da30ab08e2c9e8e23eff95643621a09c107a2e5"} Jul 16 17:36:44.504653 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.504597 2582 scope.go:117] "RemoveContainer" containerID="5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c" Jul 16 17:36:44.506323 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.506300 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" event={"ID":"cc90a598-4135-43a9-a06e-db32379f92a4","Type":"ContainerStarted","Data":"c71ce4f291cebe8bf2f03bc3617a1551ff4ba7c14852dfe6a507a25ad93cea51"} Jul 16 17:36:44.507936 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.507912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" event={"ID":"43fae792-e9bd-4de3-bd7c-8a327a7e40de","Type":"ContainerStarted","Data":"3c871467499dc65ab9e9c1cf172be6d0fb2a572263463274259c18481081a49c"} Jul 16 17:36:44.508042 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.507961 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:36:44.514754 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.514738 2582 scope.go:117] "RemoveContainer" containerID="5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c" Jul 16 17:36:44.515034 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:36:44.515011 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c\": container with ID starting with 5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c not found: ID does not exist" containerID="5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c" Jul 16 17:36:44.515091 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.515043 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c"} err="failed to get container status \"5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c\": rpc error: code = NotFound desc = could not find container \"5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c\": container with ID starting with 5e11d3fc912ac026483489ef8534559e1d9c2aed7e5cba2478b72366b588767c not found: ID does not exist" Jul 16 17:36:44.564315 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.561692 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" podStartSLOduration=1.8280749410000001 podStartE2EDuration="4.561671415s" totalImagesPullingTime="2.733596474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:44.5339937 +0000 UTC m=+466.285633258" watchObservedRunningTime="2026-07-16 17:36:44.561671415 +0000 UTC m=+466.313310975" Jul 16 17:36:44.582450 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.582401 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" podStartSLOduration=1.5780947090000002 podStartE2EDuration="5.582383408s" totalImagesPullingTime="4.004288699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:44.564813883 +0000 UTC m=+466.316453443" watchObservedRunningTime="2026-07-16 17:36:44.582383408 +0000 UTC m=+466.334023004" Jul 16 17:36:44.583362 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.583332 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-m28kv"] Jul 16 17:36:44.591377 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.591350 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-m28kv"] Jul 16 17:36:44.652830 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.652800 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5332e69-6991-44c8-811f-0786619b8fd5" path="/var/lib/kubelet/pods/a5332e69-6991-44c8-811f-0786619b8fd5/volumes" Jul 16 17:36:44.926036 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:44.926004 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:45.932472 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:45.932438 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:46.463863 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:46.463827 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-t6d4r" Jul 16 17:36:46.466022 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:46.466001 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-r59fz" Jul 16 17:36:46.521953 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:46.521725 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:46.523440 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:46.523411 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46c8gd" Jul 16 17:36:47.530786 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:47.530752 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" event={"ID":"7861aee2-fad9-4cc1-a518-268115eb3265","Type":"ContainerStarted","Data":"d849bde20a2e65125ec3e5e405fa77f7467511972df37c44de1471b97ed69533"} Jul 16 17:36:47.531174 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:47.531015 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:36:47.575054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:36:47.575001 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" podStartSLOduration=1.9086720769999999 podStartE2EDuration="7.574980386s" totalImagesPullingTime="5.666308309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:47.574160871 +0000 UTC m=+469.325800429" watchObservedRunningTime="2026-07-16 17:36:47.574980386 +0000 UTC m=+469.326619944" Jul 16 17:37:15.520536 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:15.520507 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6wn7f" Jul 16 17:37:18.537151 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:18.537119 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pp4dd" Jul 16 17:37:30.613602 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.613569 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cm4r"] Jul 16 17:37:30.614043 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.613880 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5332e69-6991-44c8-811f-0786619b8fd5" containerName="discovery" Jul 16 17:37:30.614043 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.613892 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5332e69-6991-44c8-811f-0786619b8fd5" containerName="discovery" Jul 16 17:37:30.614043 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.613941 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5332e69-6991-44c8-811f-0786619b8fd5" containerName="discovery" Jul 16 17:37:30.624206 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.624165 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" Jul 16 17:37:30.627530 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.627335 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 17:37:30.627530 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.627494 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 17:37:30.628582 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.628559 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-5mgfx\"" Jul 16 17:37:30.628735 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.628688 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cm4r"] Jul 16 17:37:30.750154 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.750121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48svq\" (UniqueName: \"kubernetes.io/projected/ef9390c2-0d47-4b87-86f1-2e7e411f9f9c-kube-api-access-48svq\") pod \"kuadrant-operator-catalog-2cm4r\" (UID: \"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c\") " pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" Jul 16 17:37:30.851582 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.851543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48svq\" (UniqueName: \"kubernetes.io/projected/ef9390c2-0d47-4b87-86f1-2e7e411f9f9c-kube-api-access-48svq\") pod \"kuadrant-operator-catalog-2cm4r\" (UID: \"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c\") " pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" Jul 16 17:37:30.861670 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.861636 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48svq\" (UniqueName: \"kubernetes.io/projected/ef9390c2-0d47-4b87-86f1-2e7e411f9f9c-kube-api-access-48svq\") pod \"kuadrant-operator-catalog-2cm4r\" (UID: \"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c\") " pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" Jul 16 17:37:30.935896 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.935813 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" Jul 16 17:37:30.975163 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:30.975112 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cm4r"] Jul 16 17:37:31.080329 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.080192 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cm4r"] Jul 16 17:37:31.082530 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:37:31.082499 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9390c2_0d47_4b87_86f1_2e7e411f9f9c.slice/crio-441972214efb18b72cadba292beb2f6785c874d84477bd2a64d4b92d8442e589 WatchSource:0}: Error finding container 441972214efb18b72cadba292beb2f6785c874d84477bd2a64d4b92d8442e589: Status 404 returned error can't find the container with id 441972214efb18b72cadba292beb2f6785c874d84477bd2a64d4b92d8442e589 Jul 16 17:37:31.187267 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.187178 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xpm4s"] Jul 16 17:37:31.191852 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.191826 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:31.200444 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.200417 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xpm4s"] Jul 16 17:37:31.355983 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.355940 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5kr2\" (UniqueName: \"kubernetes.io/projected/751610fa-ccc0-4a31-8110-e5660bb7f146-kube-api-access-m5kr2\") pod \"kuadrant-operator-catalog-xpm4s\" (UID: \"751610fa-ccc0-4a31-8110-e5660bb7f146\") " pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:31.456878 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.456793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5kr2\" (UniqueName: \"kubernetes.io/projected/751610fa-ccc0-4a31-8110-e5660bb7f146-kube-api-access-m5kr2\") pod \"kuadrant-operator-catalog-xpm4s\" (UID: \"751610fa-ccc0-4a31-8110-e5660bb7f146\") " pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:31.466921 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.466894 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5kr2\" (UniqueName: \"kubernetes.io/projected/751610fa-ccc0-4a31-8110-e5660bb7f146-kube-api-access-m5kr2\") pod \"kuadrant-operator-catalog-xpm4s\" (UID: \"751610fa-ccc0-4a31-8110-e5660bb7f146\") " pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:31.502142 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.502096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:31.646107 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.646083 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xpm4s"] Jul 16 17:37:31.647912 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:37:31.647889 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751610fa_ccc0_4a31_8110_e5660bb7f146.slice/crio-07b7528ca7e93df96c5e959d99dc878d7ebf09f8ce0e173e94e525bbed05bba6 WatchSource:0}: Error finding container 07b7528ca7e93df96c5e959d99dc878d7ebf09f8ce0e173e94e525bbed05bba6: Status 404 returned error can't find the container with id 07b7528ca7e93df96c5e959d99dc878d7ebf09f8ce0e173e94e525bbed05bba6 Jul 16 17:37:31.721323 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.721217 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" event={"ID":"751610fa-ccc0-4a31-8110-e5660bb7f146","Type":"ContainerStarted","Data":"07b7528ca7e93df96c5e959d99dc878d7ebf09f8ce0e173e94e525bbed05bba6"} Jul 16 17:37:31.722424 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:31.722390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" event={"ID":"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c","Type":"ContainerStarted","Data":"441972214efb18b72cadba292beb2f6785c874d84477bd2a64d4b92d8442e589"} Jul 16 17:37:33.735238 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:33.735147 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" event={"ID":"751610fa-ccc0-4a31-8110-e5660bb7f146","Type":"ContainerStarted","Data":"e604645ffeeea8ee4e2ab8ec9916fff9819a1a99ee29b7d1d1744fb6a66ae791"} Jul 16 17:37:33.736697 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:33.736675 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" event={"ID":"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c","Type":"ContainerStarted","Data":"edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79"} Jul 16 17:37:33.736829 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:33.736743 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" podUID="ef9390c2-0d47-4b87-86f1-2e7e411f9f9c" containerName="registry-server" containerID="cri-o://edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79" gracePeriod=2 Jul 16 17:37:33.758212 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:33.758166 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" podStartSLOduration=0.919134951 podStartE2EDuration="2.758146479s" totalImagesPullingTime="1.839011528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:37:33.756691245 +0000 UTC m=+515.508330803" watchObservedRunningTime="2026-07-16 17:37:33.758146479 +0000 UTC m=+515.509786036" Jul 16 17:37:33.776776 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:33.776728 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" podStartSLOduration=1.368069142 podStartE2EDuration="3.77671133s" totalImagesPullingTime="2.408642188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:37:33.775884806 +0000 UTC m=+515.527524363" watchObservedRunningTime="2026-07-16 17:37:33.77671133 +0000 UTC m=+515.528350925" Jul 16 17:37:33.983904 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:33.983873 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" Jul 16 17:37:34.079272 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.079235 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48svq\" (UniqueName: \"kubernetes.io/projected/ef9390c2-0d47-4b87-86f1-2e7e411f9f9c-kube-api-access-48svq\") pod \"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c\" (UID: \"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c\") " Jul 16 17:37:34.081619 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.081597 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9390c2-0d47-4b87-86f1-2e7e411f9f9c-kube-api-access-48svq" (OuterVolumeSpecName: "kube-api-access-48svq") pod "ef9390c2-0d47-4b87-86f1-2e7e411f9f9c" (UID: "ef9390c2-0d47-4b87-86f1-2e7e411f9f9c"). InnerVolumeSpecName "kube-api-access-48svq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:37:34.180389 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.180347 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48svq\" (UniqueName: \"kubernetes.io/projected/ef9390c2-0d47-4b87-86f1-2e7e411f9f9c-kube-api-access-48svq\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:34.742500 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.742465 2582 generic.go:356] "Generic (PLEG): container finished" podID="ef9390c2-0d47-4b87-86f1-2e7e411f9f9c" containerID="edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79" exitCode=0 Jul 16 17:37:34.742972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.742525 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" Jul 16 17:37:34.742972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.742561 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" event={"ID":"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c","Type":"ContainerDied","Data":"edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79"} Jul 16 17:37:34.742972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.742617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2cm4r" event={"ID":"ef9390c2-0d47-4b87-86f1-2e7e411f9f9c","Type":"ContainerDied","Data":"441972214efb18b72cadba292beb2f6785c874d84477bd2a64d4b92d8442e589"} Jul 16 17:37:34.742972 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.742646 2582 scope.go:117] "RemoveContainer" containerID="edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79" Jul 16 17:37:34.751309 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.751273 2582 scope.go:117] "RemoveContainer" containerID="edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79" Jul 16 17:37:34.751535 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:37:34.751518 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79\": container with ID starting with edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79 not found: ID does not exist" containerID="edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79" Jul 16 17:37:34.751602 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.751543 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79"} err="failed to get container status \"edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79\": rpc error: code = NotFound desc = could not find container \"edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79\": container with ID starting with edf0dd60fba3ce17c81a846f7ff41d435b48926a54ac1a540bc5e6489eccaf79 not found: ID does not exist" Jul 16 17:37:34.767969 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.767918 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cm4r"] Jul 16 17:37:34.773102 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:34.773081 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cm4r"] Jul 16 17:37:36.650768 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:36.650734 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9390c2-0d47-4b87-86f1-2e7e411f9f9c" path="/var/lib/kubelet/pods/ef9390c2-0d47-4b87-86f1-2e7e411f9f9c/volumes" Jul 16 17:37:41.503243 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:41.503189 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:41.503704 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:41.503267 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:41.525932 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:41.525905 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:41.795399 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:41.795374 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xpm4s" Jul 16 17:37:46.073978 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.073945 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc"] Jul 16 17:37:46.074398 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.074252 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9390c2-0d47-4b87-86f1-2e7e411f9f9c" containerName="registry-server" Jul 16 17:37:46.074398 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.074262 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9390c2-0d47-4b87-86f1-2e7e411f9f9c" containerName="registry-server" Jul 16 17:37:46.074398 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.074345 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9390c2-0d47-4b87-86f1-2e7e411f9f9c" containerName="registry-server" Jul 16 17:37:46.078665 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.078648 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.085255 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.085233 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2tmqz\"" Jul 16 17:37:46.097461 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.097428 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc"] Jul 16 17:37:46.170881 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.170843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.171057 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.170890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.171057 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.170964 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgjx\" (UniqueName: \"kubernetes.io/projected/ac4202e8-0b36-4229-8fa9-ab8784ff48df-kube-api-access-gsgjx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.271652 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.271611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.271652 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.271660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.271838 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.271691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgjx\" (UniqueName: \"kubernetes.io/projected/ac4202e8-0b36-4229-8fa9-ab8784ff48df-kube-api-access-gsgjx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.272085 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.272061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.272149 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.272071 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.281937 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.281900 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgjx\" (UniqueName: \"kubernetes.io/projected/ac4202e8-0b36-4229-8fa9-ab8784ff48df-kube-api-access-gsgjx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.388023 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.387925 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:46.525637 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.525616 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc"] Jul 16 17:37:46.528143 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:37:46.528115 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4202e8_0b36_4229_8fa9_ab8784ff48df.slice/crio-d1b7dc133e6fd557400a60b24a86b33e8d61c69edacd29245c449589937c0c90 WatchSource:0}: Error finding container d1b7dc133e6fd557400a60b24a86b33e8d61c69edacd29245c449589937c0c90: Status 404 returned error can't find the container with id d1b7dc133e6fd557400a60b24a86b33e8d61c69edacd29245c449589937c0c90 Jul 16 17:37:46.650993 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.650919 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs"] Jul 16 17:37:46.654193 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.654176 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.668029 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.667998 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs"] Jul 16 17:37:46.675490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.675462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.675632 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.675509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.675632 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.675560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwchl\" (UniqueName: \"kubernetes.io/projected/029c6c08-d801-442e-a164-b007387c92bd-kube-api-access-rwchl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.775941 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.775899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwchl\" (UniqueName: \"kubernetes.io/projected/029c6c08-d801-442e-a164-b007387c92bd-kube-api-access-rwchl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.776126 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.775969 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.776126 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.776004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.776453 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.776434 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.776497 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.776464 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.786884 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.786855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwchl\" (UniqueName: \"kubernetes.io/projected/029c6c08-d801-442e-a164-b007387c92bd-kube-api-access-rwchl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:46.794888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.794860 2582 generic.go:356] "Generic (PLEG): container finished" podID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerID="d5c98cec242e581eb4d2952d76ce7051aa906657b9b263e12157a4619be3bcbf" exitCode=0 Jul 16 17:37:46.795024 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.794911 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" event={"ID":"ac4202e8-0b36-4229-8fa9-ab8784ff48df","Type":"ContainerDied","Data":"d5c98cec242e581eb4d2952d76ce7051aa906657b9b263e12157a4619be3bcbf"} Jul 16 17:37:46.795024 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:46.794946 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" event={"ID":"ac4202e8-0b36-4229-8fa9-ab8784ff48df","Type":"ContainerStarted","Data":"d1b7dc133e6fd557400a60b24a86b33e8d61c69edacd29245c449589937c0c90"} Jul 16 17:37:47.005331 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.005226 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:47.088930 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.088901 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd"] Jul 16 17:37:47.093712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.093688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.143079 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.143040 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd"] Jul 16 17:37:47.179637 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.179599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.179831 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.179727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.179831 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.179761 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv277\" (UniqueName: \"kubernetes.io/projected/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-kube-api-access-wv277\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.187025 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:37:47.186997 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029c6c08_d801_442e_a164_b007387c92bd.slice/crio-47dd599e0de0553ecb5d346a0f8315ddc3e56274be6c9f26122bf38da7e69245 WatchSource:0}: Error finding container 47dd599e0de0553ecb5d346a0f8315ddc3e56274be6c9f26122bf38da7e69245: Status 404 returned error can't find the container with id 47dd599e0de0553ecb5d346a0f8315ddc3e56274be6c9f26122bf38da7e69245 Jul 16 17:37:47.188348 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.188323 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs"] Jul 16 17:37:47.280461 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.280427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.280618 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.280502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.280618 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.280520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv277\" (UniqueName: \"kubernetes.io/projected/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-kube-api-access-wv277\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.280890 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.280864 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.280931 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.280884 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.292885 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.292861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv277\" (UniqueName: \"kubernetes.io/projected/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-kube-api-access-wv277\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.410658 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.410621 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:47.497527 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.497188 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6"] Jul 16 17:37:47.502436 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.502407 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.525766 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.525730 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6"] Jul 16 17:37:47.571362 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.571334 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd"] Jul 16 17:37:47.583253 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.583228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554h2\" (UniqueName: \"kubernetes.io/projected/ae59f884-9a62-4b89-8570-fe266763d135-kube-api-access-554h2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.583376 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.583274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.583376 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.583348 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.683821 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.683785 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554h2\" (UniqueName: \"kubernetes.io/projected/ae59f884-9a62-4b89-8570-fe266763d135-kube-api-access-554h2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.683963 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.683832 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.683963 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.683873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.684252 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.684234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.684310 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.684264 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.694977 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.694954 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-554h2\" (UniqueName: \"kubernetes.io/projected/ae59f884-9a62-4b89-8570-fe266763d135-kube-api-access-554h2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.801096 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.801066 2582 generic.go:356] "Generic (PLEG): container finished" podID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerID="8b5ecd2a1f7a005e124775daee043c6b0936498d5d832d260c969cabf6c707e2" exitCode=0 Jul 16 17:37:47.801262 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.801152 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" event={"ID":"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46","Type":"ContainerDied","Data":"8b5ecd2a1f7a005e124775daee043c6b0936498d5d832d260c969cabf6c707e2"} Jul 16 17:37:47.801262 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.801206 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" event={"ID":"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46","Type":"ContainerStarted","Data":"c4c8269731c47e67cc0577ad6731cd123b4991cb69a9bfa4e44dad4a271daf01"} Jul 16 17:37:47.802843 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.802820 2582 generic.go:356] "Generic (PLEG): container finished" podID="029c6c08-d801-442e-a164-b007387c92bd" containerID="adc772c87e7d6513f405c97e68602dfe727a7ec13cc0609f44568f8ed435e5ad" exitCode=0 Jul 16 17:37:47.802953 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.802879 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" event={"ID":"029c6c08-d801-442e-a164-b007387c92bd","Type":"ContainerDied","Data":"adc772c87e7d6513f405c97e68602dfe727a7ec13cc0609f44568f8ed435e5ad"} Jul 16 17:37:47.802953 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.802912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" event={"ID":"029c6c08-d801-442e-a164-b007387c92bd","Type":"ContainerStarted","Data":"47dd599e0de0553ecb5d346a0f8315ddc3e56274be6c9f26122bf38da7e69245"} Jul 16 17:37:47.813142 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.813118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:47.973493 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:47.973464 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6"] Jul 16 17:37:47.975203 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:37:47.975176 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae59f884_9a62_4b89_8570_fe266763d135.slice/crio-600241177f152be3f5563531467cc708827178876abe0750dec7777b48bc2e79 WatchSource:0}: Error finding container 600241177f152be3f5563531467cc708827178876abe0750dec7777b48bc2e79: Status 404 returned error can't find the container with id 600241177f152be3f5563531467cc708827178876abe0750dec7777b48bc2e79 Jul 16 17:37:48.810081 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:48.810049 2582 generic.go:356] "Generic (PLEG): container finished" podID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerID="dbc89fa3887d10b75184a25cf9d6b391eb56eed6722dc8ae84508205b793b144" exitCode=0 Jul 16 17:37:48.810516 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:48.810130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" event={"ID":"ac4202e8-0b36-4229-8fa9-ab8784ff48df","Type":"ContainerDied","Data":"dbc89fa3887d10b75184a25cf9d6b391eb56eed6722dc8ae84508205b793b144"} Jul 16 17:37:48.811664 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:48.811642 2582 generic.go:356] "Generic (PLEG): container finished" podID="ae59f884-9a62-4b89-8570-fe266763d135" containerID="f369f2540443af39d3d3b80880e34bae8a82c5e0ad7aa81a25fec944a00b6f5d" exitCode=0 Jul 16 17:37:48.811806 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:48.811777 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" event={"ID":"ae59f884-9a62-4b89-8570-fe266763d135","Type":"ContainerDied","Data":"f369f2540443af39d3d3b80880e34bae8a82c5e0ad7aa81a25fec944a00b6f5d"} Jul 16 17:37:48.811881 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:48.811835 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" event={"ID":"ae59f884-9a62-4b89-8570-fe266763d135","Type":"ContainerStarted","Data":"600241177f152be3f5563531467cc708827178876abe0750dec7777b48bc2e79"} Jul 16 17:37:48.813634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:48.813612 2582 generic.go:356] "Generic (PLEG): container finished" podID="029c6c08-d801-442e-a164-b007387c92bd" containerID="f06d31998ebf93183f81e26972b023b609ae593c34c1eadcc41d9c1959540330" exitCode=0 Jul 16 17:37:48.813704 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:48.813672 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" event={"ID":"029c6c08-d801-442e-a164-b007387c92bd","Type":"ContainerDied","Data":"f06d31998ebf93183f81e26972b023b609ae593c34c1eadcc41d9c1959540330"} Jul 16 17:37:49.820068 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:49.820031 2582 generic.go:356] "Generic (PLEG): container finished" podID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerID="64645761f9ebe911fef59758fd31e2e68e0cc623b7038545a0e85cfa6b2da5b0" exitCode=0 Jul 16 17:37:49.820516 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:49.820114 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" event={"ID":"ac4202e8-0b36-4229-8fa9-ab8784ff48df","Type":"ContainerDied","Data":"64645761f9ebe911fef59758fd31e2e68e0cc623b7038545a0e85cfa6b2da5b0"} Jul 16 17:37:49.821909 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:49.821889 2582 generic.go:356] "Generic (PLEG): container finished" podID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerID="cce03bf1cdca659d6b31412032f228b98e28aa51df44ccd3efa1f833d3699658" exitCode=0 Jul 16 17:37:49.821983 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:49.821957 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" event={"ID":"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46","Type":"ContainerDied","Data":"cce03bf1cdca659d6b31412032f228b98e28aa51df44ccd3efa1f833d3699658"} Jul 16 17:37:49.823868 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:49.823844 2582 generic.go:356] "Generic (PLEG): container finished" podID="ae59f884-9a62-4b89-8570-fe266763d135" containerID="c7d12dbe2c3c3d02f38809fcc0567034fb5d36a84b8f17934c033f3e61fe661f" exitCode=0 Jul 16 17:37:49.823973 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:49.823927 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" event={"ID":"ae59f884-9a62-4b89-8570-fe266763d135","Type":"ContainerDied","Data":"c7d12dbe2c3c3d02f38809fcc0567034fb5d36a84b8f17934c033f3e61fe661f"} Jul 16 17:37:49.826086 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:49.826065 2582 generic.go:356] "Generic (PLEG): container finished" podID="029c6c08-d801-442e-a164-b007387c92bd" containerID="71c12f098f92002c6898956c34b03a487a331ea3d82636d77e67fa86417fa8e0" exitCode=0 Jul 16 17:37:49.826158 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:49.826122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" event={"ID":"029c6c08-d801-442e-a164-b007387c92bd","Type":"ContainerDied","Data":"71c12f098f92002c6898956c34b03a487a331ea3d82636d77e67fa86417fa8e0"} Jul 16 17:37:50.832417 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:50.832384 2582 generic.go:356] "Generic (PLEG): container finished" podID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerID="878b6772a04766cdb19abe9ea05efd62396be9cb456748ae5129088078940029" exitCode=0 Jul 16 17:37:50.832886 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:50.832464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" event={"ID":"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46","Type":"ContainerDied","Data":"878b6772a04766cdb19abe9ea05efd62396be9cb456748ae5129088078940029"} Jul 16 17:37:50.834442 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:50.834417 2582 generic.go:356] "Generic (PLEG): container finished" podID="ae59f884-9a62-4b89-8570-fe266763d135" containerID="af7247203e8466f918fe5c19487dc91c8362efcd46f59e973e8e2071b15a4064" exitCode=0 Jul 16 17:37:50.834576 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:50.834464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" event={"ID":"ae59f884-9a62-4b89-8570-fe266763d135","Type":"ContainerDied","Data":"af7247203e8466f918fe5c19487dc91c8362efcd46f59e973e8e2071b15a4064"} Jul 16 17:37:50.970751 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:50.970726 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:51.008542 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.008514 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:51.010699 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.010675 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-util\") pod \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " Jul 16 17:37:51.010831 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.010713 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsgjx\" (UniqueName: \"kubernetes.io/projected/ac4202e8-0b36-4229-8fa9-ab8784ff48df-kube-api-access-gsgjx\") pod \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " Jul 16 17:37:51.010831 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.010746 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-bundle\") pod \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\" (UID: \"ac4202e8-0b36-4229-8fa9-ab8784ff48df\") " Jul 16 17:37:51.011257 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.011234 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-bundle" (OuterVolumeSpecName: "bundle") pod "ac4202e8-0b36-4229-8fa9-ab8784ff48df" (UID: "ac4202e8-0b36-4229-8fa9-ab8784ff48df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:51.012917 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.012895 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4202e8-0b36-4229-8fa9-ab8784ff48df-kube-api-access-gsgjx" (OuterVolumeSpecName: "kube-api-access-gsgjx") pod "ac4202e8-0b36-4229-8fa9-ab8784ff48df" (UID: "ac4202e8-0b36-4229-8fa9-ab8784ff48df"). InnerVolumeSpecName "kube-api-access-gsgjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:37:51.020234 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.020203 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-util" (OuterVolumeSpecName: "util") pod "ac4202e8-0b36-4229-8fa9-ab8784ff48df" (UID: "ac4202e8-0b36-4229-8fa9-ab8784ff48df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:51.111316 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.111198 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwchl\" (UniqueName: \"kubernetes.io/projected/029c6c08-d801-442e-a164-b007387c92bd-kube-api-access-rwchl\") pod \"029c6c08-d801-442e-a164-b007387c92bd\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " Jul 16 17:37:51.111316 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.111273 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-bundle\") pod \"029c6c08-d801-442e-a164-b007387c92bd\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " Jul 16 17:37:51.111537 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.111328 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-util\") pod \"029c6c08-d801-442e-a164-b007387c92bd\" (UID: \"029c6c08-d801-442e-a164-b007387c92bd\") " Jul 16 17:37:51.111537 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.111459 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-util\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:51.111537 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.111474 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsgjx\" (UniqueName: \"kubernetes.io/projected/ac4202e8-0b36-4229-8fa9-ab8784ff48df-kube-api-access-gsgjx\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:51.111537 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.111486 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac4202e8-0b36-4229-8fa9-ab8784ff48df-bundle\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:51.111819 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.111787 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-bundle" (OuterVolumeSpecName: "bundle") pod "029c6c08-d801-442e-a164-b007387c92bd" (UID: "029c6c08-d801-442e-a164-b007387c92bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:51.113447 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.113423 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029c6c08-d801-442e-a164-b007387c92bd-kube-api-access-rwchl" (OuterVolumeSpecName: "kube-api-access-rwchl") pod "029c6c08-d801-442e-a164-b007387c92bd" (UID: "029c6c08-d801-442e-a164-b007387c92bd"). InnerVolumeSpecName "kube-api-access-rwchl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:37:51.116792 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.116764 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-util" (OuterVolumeSpecName: "util") pod "029c6c08-d801-442e-a164-b007387c92bd" (UID: "029c6c08-d801-442e-a164-b007387c92bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:51.212436 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.212410 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwchl\" (UniqueName: \"kubernetes.io/projected/029c6c08-d801-442e-a164-b007387c92bd-kube-api-access-rwchl\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:51.212436 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.212434 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-bundle\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:51.212436 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.212444 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029c6c08-d801-442e-a164-b007387c92bd-util\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:51.845087 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.845051 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" Jul 16 17:37:51.845087 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.845063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs" event={"ID":"029c6c08-d801-442e-a164-b007387c92bd","Type":"ContainerDied","Data":"47dd599e0de0553ecb5d346a0f8315ddc3e56274be6c9f26122bf38da7e69245"} Jul 16 17:37:51.845543 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.845110 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47dd599e0de0553ecb5d346a0f8315ddc3e56274be6c9f26122bf38da7e69245" Jul 16 17:37:51.846936 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.846919 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" Jul 16 17:37:51.847046 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.846951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc" event={"ID":"ac4202e8-0b36-4229-8fa9-ab8784ff48df","Type":"ContainerDied","Data":"d1b7dc133e6fd557400a60b24a86b33e8d61c69edacd29245c449589937c0c90"} Jul 16 17:37:51.847046 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:51.846990 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b7dc133e6fd557400a60b24a86b33e8d61c69edacd29245c449589937c0c90" Jul 16 17:37:52.020470 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.020443 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:37:52.023160 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.023141 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:52.120467 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.120369 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-util\") pod \"ae59f884-9a62-4b89-8570-fe266763d135\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " Jul 16 17:37:52.120467 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.120420 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv277\" (UniqueName: \"kubernetes.io/projected/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-kube-api-access-wv277\") pod \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " Jul 16 17:37:52.120467 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.120446 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-bundle\") pod \"ae59f884-9a62-4b89-8570-fe266763d135\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " Jul 16 17:37:52.120725 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.120471 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-util\") pod \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " Jul 16 17:37:52.120725 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.120534 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-bundle\") pod \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\" (UID: \"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46\") " Jul 16 17:37:52.120725 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.120559 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-554h2\" (UniqueName: \"kubernetes.io/projected/ae59f884-9a62-4b89-8570-fe266763d135-kube-api-access-554h2\") pod \"ae59f884-9a62-4b89-8570-fe266763d135\" (UID: \"ae59f884-9a62-4b89-8570-fe266763d135\") " Jul 16 17:37:52.121044 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.121017 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-bundle" (OuterVolumeSpecName: "bundle") pod "77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" (UID: "77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:52.121219 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.121190 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-bundle" (OuterVolumeSpecName: "bundle") pod "ae59f884-9a62-4b89-8570-fe266763d135" (UID: "ae59f884-9a62-4b89-8570-fe266763d135"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:52.122765 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.122734 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-kube-api-access-wv277" (OuterVolumeSpecName: "kube-api-access-wv277") pod "77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" (UID: "77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46"). InnerVolumeSpecName "kube-api-access-wv277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:37:52.122858 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.122789 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae59f884-9a62-4b89-8570-fe266763d135-kube-api-access-554h2" (OuterVolumeSpecName: "kube-api-access-554h2") pod "ae59f884-9a62-4b89-8570-fe266763d135" (UID: "ae59f884-9a62-4b89-8570-fe266763d135"). InnerVolumeSpecName "kube-api-access-554h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:37:52.126507 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.126488 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-util" (OuterVolumeSpecName: "util") pod "ae59f884-9a62-4b89-8570-fe266763d135" (UID: "ae59f884-9a62-4b89-8570-fe266763d135"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:52.128331 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.128309 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-util" (OuterVolumeSpecName: "util") pod "77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" (UID: "77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:52.221304 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.221246 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-bundle\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:52.221304 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.221275 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554h2\" (UniqueName: \"kubernetes.io/projected/ae59f884-9a62-4b89-8570-fe266763d135-kube-api-access-554h2\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:52.221500 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.221314 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-util\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:52.221500 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.221327 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv277\" (UniqueName: \"kubernetes.io/projected/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-kube-api-access-wv277\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:52.221500 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.221338 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae59f884-9a62-4b89-8570-fe266763d135-bundle\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:52.221500 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.221351 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46-util\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:37:52.853986 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.853954 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" Jul 16 17:37:52.854375 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.853947 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd" event={"ID":"77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46","Type":"ContainerDied","Data":"c4c8269731c47e67cc0577ad6731cd123b4991cb69a9bfa4e44dad4a271daf01"} Jul 16 17:37:52.854375 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.854123 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4c8269731c47e67cc0577ad6731cd123b4991cb69a9bfa4e44dad4a271daf01" Jul 16 17:37:52.856108 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.856075 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" event={"ID":"ae59f884-9a62-4b89-8570-fe266763d135","Type":"ContainerDied","Data":"600241177f152be3f5563531467cc708827178876abe0750dec7777b48bc2e79"} Jul 16 17:37:52.856230 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.856128 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="600241177f152be3f5563531467cc708827178876abe0750dec7777b48bc2e79" Jul 16 17:37:52.856230 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:37:52.856085 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6" Jul 16 17:38:01.937219 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937185 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb"] Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937497 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerName="extract" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937508 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerName="extract" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937519 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerName="util" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937524 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerName="util" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937531 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029c6c08-d801-442e-a164-b007387c92bd" containerName="extract" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937536 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="029c6c08-d801-442e-a164-b007387c92bd" containerName="extract" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937542 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae59f884-9a62-4b89-8570-fe266763d135" containerName="pull" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937548 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae59f884-9a62-4b89-8570-fe266763d135" containerName="pull" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937554 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerName="pull" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937559 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerName="pull" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937569 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae59f884-9a62-4b89-8570-fe266763d135" containerName="util" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937575 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae59f884-9a62-4b89-8570-fe266763d135" containerName="util" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937582 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029c6c08-d801-442e-a164-b007387c92bd" containerName="pull" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937586 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="029c6c08-d801-442e-a164-b007387c92bd" containerName="pull" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937592 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerName="pull" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937597 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerName="pull" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937605 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerName="util" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937610 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerName="util" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937615 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae59f884-9a62-4b89-8570-fe266763d135" containerName="extract" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937620 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae59f884-9a62-4b89-8570-fe266763d135" containerName="extract" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937628 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029c6c08-d801-442e-a164-b007387c92bd" containerName="util" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937633 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="029c6c08-d801-442e-a164-b007387c92bd" containerName="util" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937638 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerName="extract" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937643 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerName="extract" Jul 16 17:38:01.937681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937690 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="029c6c08-d801-442e-a164-b007387c92bd" containerName="extract" Jul 16 17:38:01.938409 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937698 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46" containerName="extract" Jul 16 17:38:01.938409 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937704 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae59f884-9a62-4b89-8570-fe266763d135" containerName="extract" Jul 16 17:38:01.938409 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.937711 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac4202e8-0b36-4229-8fa9-ab8784ff48df" containerName="extract" Jul 16 17:38:01.942167 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.942144 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" Jul 16 17:38:01.946051 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.946027 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 16 17:38:01.946051 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.946048 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-t4l7m\"" Jul 16 17:38:01.953661 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:01.953636 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb"] Jul 16 17:38:02.008979 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:02.008939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkk42\" (UniqueName: \"kubernetes.io/projected/a478a22d-09f4-4a8e-8bf7-b18673b760e6-kube-api-access-gkk42\") pod \"dns-operator-controller-manager-864dd78c4c-nqwqb\" (UID: \"a478a22d-09f4-4a8e-8bf7-b18673b760e6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" Jul 16 17:38:02.109866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:02.109823 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkk42\" (UniqueName: \"kubernetes.io/projected/a478a22d-09f4-4a8e-8bf7-b18673b760e6-kube-api-access-gkk42\") pod \"dns-operator-controller-manager-864dd78c4c-nqwqb\" (UID: \"a478a22d-09f4-4a8e-8bf7-b18673b760e6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" Jul 16 17:38:02.136086 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:02.136043 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkk42\" (UniqueName: \"kubernetes.io/projected/a478a22d-09f4-4a8e-8bf7-b18673b760e6-kube-api-access-gkk42\") pod \"dns-operator-controller-manager-864dd78c4c-nqwqb\" (UID: \"a478a22d-09f4-4a8e-8bf7-b18673b760e6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" Jul 16 17:38:02.253008 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:02.252906 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" Jul 16 17:38:02.399234 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:02.399210 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb"] Jul 16 17:38:02.401024 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:38:02.400993 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda478a22d_09f4_4a8e_8bf7_b18673b760e6.slice/crio-63dc885afb654a6a7bcdc8ff0a79143beef0678cddba8bc5a5191f76109e7056 WatchSource:0}: Error finding container 63dc885afb654a6a7bcdc8ff0a79143beef0678cddba8bc5a5191f76109e7056: Status 404 returned error can't find the container with id 63dc885afb654a6a7bcdc8ff0a79143beef0678cddba8bc5a5191f76109e7056 Jul 16 17:38:02.906093 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:02.906059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" event={"ID":"a478a22d-09f4-4a8e-8bf7-b18673b760e6","Type":"ContainerStarted","Data":"63dc885afb654a6a7bcdc8ff0a79143beef0678cddba8bc5a5191f76109e7056"} Jul 16 17:38:04.919569 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:04.919525 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" event={"ID":"a478a22d-09f4-4a8e-8bf7-b18673b760e6","Type":"ContainerStarted","Data":"b1e5fc22bf41fc0f0100182433cdfb1a24d8b7db60715a640098d0bec8fff732"} Jul 16 17:38:04.919945 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:04.919597 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" Jul 16 17:38:04.943827 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:04.943729 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" podStartSLOduration=1.6592056670000002 podStartE2EDuration="3.943714378s" totalImagesPullingTime="2.284508711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:04.942085324 +0000 UTC m=+546.693724881" watchObservedRunningTime="2026-07-16 17:38:04.943714378 +0000 UTC m=+546.695353936" Jul 16 17:38:05.564936 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:05.564896 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv"] Jul 16 17:38:05.568404 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:05.568371 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:05.572595 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:05.572576 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wjhz9\"" Jul 16 17:38:05.586061 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:05.586033 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv"] Jul 16 17:38:05.641759 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:05.641726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/692d87fa-192c-4372-a34d-a5e6fe4d8b09-kube-api-access-g4jdj\") pod \"limitador-operator-controller-manager-859d7b78bd-vjbdv\" (UID: \"692d87fa-192c-4372-a34d-a5e6fe4d8b09\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:05.742681 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:05.742644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/692d87fa-192c-4372-a34d-a5e6fe4d8b09-kube-api-access-g4jdj\") pod \"limitador-operator-controller-manager-859d7b78bd-vjbdv\" (UID: \"692d87fa-192c-4372-a34d-a5e6fe4d8b09\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:05.756026 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:05.756002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/692d87fa-192c-4372-a34d-a5e6fe4d8b09-kube-api-access-g4jdj\") pod \"limitador-operator-controller-manager-859d7b78bd-vjbdv\" (UID: \"692d87fa-192c-4372-a34d-a5e6fe4d8b09\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:05.879075 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:05.878993 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:06.026093 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:06.026072 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv"] Jul 16 17:38:06.027911 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:38:06.027885 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692d87fa_192c_4372_a34d_a5e6fe4d8b09.slice/crio-8a80131e76ba47b02db51e879e1fcadee244dcca76f96cf0b0e258f1c8216286 WatchSource:0}: Error finding container 8a80131e76ba47b02db51e879e1fcadee244dcca76f96cf0b0e258f1c8216286: Status 404 returned error can't find the container with id 8a80131e76ba47b02db51e879e1fcadee244dcca76f96cf0b0e258f1c8216286 Jul 16 17:38:06.934251 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:06.934213 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" event={"ID":"692d87fa-192c-4372-a34d-a5e6fe4d8b09","Type":"ContainerStarted","Data":"8a80131e76ba47b02db51e879e1fcadee244dcca76f96cf0b0e258f1c8216286"} Jul 16 17:38:07.941128 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:07.941084 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" event={"ID":"692d87fa-192c-4372-a34d-a5e6fe4d8b09","Type":"ContainerStarted","Data":"c0c5805fb4afe8ec74441dfba516a7c723d76e48ef6b6ebcda2a49a9f16fc9e0"} Jul 16 17:38:07.941515 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:07.941179 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:07.962671 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:07.962582 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" podStartSLOduration=1.279083947 podStartE2EDuration="2.962566524s" totalImagesPullingTime="1.683482577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:07.961485905 +0000 UTC m=+549.713125464" watchObservedRunningTime="2026-07-16 17:38:07.962566524 +0000 UTC m=+549.714206082" Jul 16 17:38:10.610461 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.610419 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6"] Jul 16 17:38:10.614005 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.613989 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:10.618412 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.618392 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8cgsj\"" Jul 16 17:38:10.651114 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.651085 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6"] Jul 16 17:38:10.685541 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.685507 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4rw\" (UniqueName: \"kubernetes.io/projected/2f4f10b1-a8b5-46fb-abae-29e595045d0b-kube-api-access-4d4rw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" (UID: \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:10.685707 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.685547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f4f10b1-a8b5-46fb-abae-29e595045d0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" (UID: \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:10.786374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.786336 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4rw\" (UniqueName: \"kubernetes.io/projected/2f4f10b1-a8b5-46fb-abae-29e595045d0b-kube-api-access-4d4rw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" (UID: \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:10.786374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.786387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f4f10b1-a8b5-46fb-abae-29e595045d0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" (UID: \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:10.786765 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.786737 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f4f10b1-a8b5-46fb-abae-29e595045d0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" (UID: \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:10.816912 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.816881 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4rw\" (UniqueName: \"kubernetes.io/projected/2f4f10b1-a8b5-46fb-abae-29e595045d0b-kube-api-access-4d4rw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" (UID: \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:10.923790 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:10.923705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:11.070507 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:11.070483 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6"] Jul 16 17:38:11.072405 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:38:11.072376 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4f10b1_a8b5_46fb_abae_29e595045d0b.slice/crio-1bf400124386cd97a9a1b0758f13c91667758da167c4cffec96985bae4aa46d1 WatchSource:0}: Error finding container 1bf400124386cd97a9a1b0758f13c91667758da167c4cffec96985bae4aa46d1: Status 404 returned error can't find the container with id 1bf400124386cd97a9a1b0758f13c91667758da167c4cffec96985bae4aa46d1 Jul 16 17:38:11.965038 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:11.965001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" event={"ID":"2f4f10b1-a8b5-46fb-abae-29e595045d0b","Type":"ContainerStarted","Data":"1bf400124386cd97a9a1b0758f13c91667758da167c4cffec96985bae4aa46d1"} Jul 16 17:38:14.545075 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:14.545040 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-s87j6"] Jul 16 17:38:14.548465 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:14.548438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" Jul 16 17:38:14.551194 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:14.551173 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zfr82\"" Jul 16 17:38:14.559703 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:14.559674 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-s87j6"] Jul 16 17:38:14.619647 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:14.619615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdwt\" (UniqueName: \"kubernetes.io/projected/7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97-kube-api-access-tjdwt\") pod \"authorino-operator-75cc8d5cb7-s87j6\" (UID: \"7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" Jul 16 17:38:14.720828 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:14.720781 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdwt\" (UniqueName: \"kubernetes.io/projected/7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97-kube-api-access-tjdwt\") pod \"authorino-operator-75cc8d5cb7-s87j6\" (UID: \"7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" Jul 16 17:38:14.731612 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:14.731577 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdwt\" (UniqueName: \"kubernetes.io/projected/7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97-kube-api-access-tjdwt\") pod \"authorino-operator-75cc8d5cb7-s87j6\" (UID: \"7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" Jul 16 17:38:14.862976 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:14.862883 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" Jul 16 17:38:15.029889 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:15.029859 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-s87j6"] Jul 16 17:38:15.031647 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:38:15.031618 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7c2a80_7c0b_48e2_b52d_ac1e0facdc97.slice/crio-8bf3d33b841606ed57fd21525eebeec50b338fc2bd6025b544cddb0608f56bd6 WatchSource:0}: Error finding container 8bf3d33b841606ed57fd21525eebeec50b338fc2bd6025b544cddb0608f56bd6: Status 404 returned error can't find the container with id 8bf3d33b841606ed57fd21525eebeec50b338fc2bd6025b544cddb0608f56bd6 Jul 16 17:38:15.927867 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:15.927835 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nqwqb" Jul 16 17:38:15.990894 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:15.990847 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" event={"ID":"7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97","Type":"ContainerStarted","Data":"8bf3d33b841606ed57fd21525eebeec50b338fc2bd6025b544cddb0608f56bd6"} Jul 16 17:38:18.948866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:18.948828 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:19.009818 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:19.009774 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" event={"ID":"2f4f10b1-a8b5-46fb-abae-29e595045d0b","Type":"ContainerStarted","Data":"6f82f51fc4eadae3fe2f2df3069e7c5947c48b0d56a9133b154abb9a39b90750"} Jul 16 17:38:19.009986 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:19.009857 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:19.011604 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:19.011574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" event={"ID":"7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97","Type":"ContainerStarted","Data":"ba19d04e93bdd88cd273874acfe81019e25036955557cf2e78fd19ba4dd65bdc"} Jul 16 17:38:19.011716 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:19.011670 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" Jul 16 17:38:19.031027 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:19.030927 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" podStartSLOduration=1.321479712 podStartE2EDuration="9.030908705s" totalImagesPullingTime="7.709428993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:19.029881445 +0000 UTC m=+560.781521003" watchObservedRunningTime="2026-07-16 17:38:19.030908705 +0000 UTC m=+560.782548262" Jul 16 17:38:19.047887 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:19.047824 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" podStartSLOduration=1.3003234670000001 podStartE2EDuration="5.047794726s" totalImagesPullingTime="3.747471259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:19.047388208 +0000 UTC m=+560.799027786" watchObservedRunningTime="2026-07-16 17:38:19.047794726 +0000 UTC m=+560.799434286" Jul 16 17:38:30.018653 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:30.018621 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s87j6" Jul 16 17:38:30.019004 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:30.018689 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:31.869516 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.869476 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6"] Jul 16 17:38:31.869958 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.869689 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" containerName="manager" containerID="cri-o://6f82f51fc4eadae3fe2f2df3069e7c5947c48b0d56a9133b154abb9a39b90750" gracePeriod=2 Jul 16 17:38:31.879058 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.878832 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6"] Jul 16 17:38:31.889256 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:31.889211 2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2f4f10b1-a8b5-46fb-abae-29e595045d0b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\": pods \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:31.894408 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.894382 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv"] Jul 16 17:38:31.894587 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.894566 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" containerName="manager" containerID="cri-o://c0c5805fb4afe8ec74441dfba516a7c723d76e48ef6b6ebcda2a49a9f16fc9e0" gracePeriod=2 Jul 16 17:38:31.897053 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:31.897018 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:31.903423 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.903397 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8"] Jul 16 17:38:31.903736 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.903725 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" containerName="manager" Jul 16 17:38:31.903784 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.903738 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" containerName="manager" Jul 16 17:38:31.903823 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.903805 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" containerName="manager" Jul 16 17:38:31.906916 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.906880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:31.910198 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.910171 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv"] Jul 16 17:38:31.927570 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.927541 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8"] Jul 16 17:38:31.928818 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:31.928784 2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"692d87fa-192c-4372-a34d-a5e6fe4d8b09\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-vjbdv\": pods \"limitador-operator-controller-manager-859d7b78bd-vjbdv\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:31.931758 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:31.931492 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vjbdv\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:31.932742 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.932718 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl"] Jul 16 17:38:31.933422 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.933181 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" containerName="manager" Jul 16 17:38:31.933422 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.933202 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" containerName="manager" Jul 16 17:38:31.933422 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.933329 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" containerName="manager" Jul 16 17:38:31.936475 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.936455 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" Jul 16 17:38:31.936720 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:31.936695 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:31.956390 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:31.956355 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl"] Jul 16 17:38:31.967589 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:31.967551 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:31.970144 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:31.970115 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vjbdv\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:32.080142 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.080112 2582 generic.go:356] "Generic (PLEG): container finished" podID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" containerID="c0c5805fb4afe8ec74441dfba516a7c723d76e48ef6b6ebcda2a49a9f16fc9e0" exitCode=0 Jul 16 17:38:32.082054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.082030 2582 generic.go:356] "Generic (PLEG): container finished" podID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" containerID="6f82f51fc4eadae3fe2f2df3069e7c5947c48b0d56a9133b154abb9a39b90750" exitCode=0 Jul 16 17:38:32.082190 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.082064 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdx7\" (UniqueName: \"kubernetes.io/projected/9ff6b759-d3d3-4d1e-b216-02a81309f907-kube-api-access-qvdx7\") pod \"limitador-operator-controller-manager-859d7b78bd-jszsl\" (UID: \"9ff6b759-d3d3-4d1e-b216-02a81309f907\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" Jul 16 17:38:32.082190 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.082119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcxv\" (UniqueName: \"kubernetes.io/projected/b02109e0-dfcd-4633-9b1d-4d037c55c073-kube-api-access-nwcxv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n26v8\" (UID: \"b02109e0-dfcd-4633-9b1d-4d037c55c073\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:32.082307 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.082203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b02109e0-dfcd-4633-9b1d-4d037c55c073-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n26v8\" (UID: \"b02109e0-dfcd-4633-9b1d-4d037c55c073\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:32.138714 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.138691 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:32.141290 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:32.141246 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vjbdv\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:32.145791 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:32.145764 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:32.162619 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.162592 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:32.165229 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:32.165202 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vjbdv\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:32.167591 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:38:32.167569 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4qpl6\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:32.182995 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.182966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdx7\" (UniqueName: \"kubernetes.io/projected/9ff6b759-d3d3-4d1e-b216-02a81309f907-kube-api-access-qvdx7\") pod \"limitador-operator-controller-manager-859d7b78bd-jszsl\" (UID: \"9ff6b759-d3d3-4d1e-b216-02a81309f907\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" Jul 16 17:38:32.183086 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.183016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcxv\" (UniqueName: \"kubernetes.io/projected/b02109e0-dfcd-4633-9b1d-4d037c55c073-kube-api-access-nwcxv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n26v8\" (UID: \"b02109e0-dfcd-4633-9b1d-4d037c55c073\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:32.183150 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.183089 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b02109e0-dfcd-4633-9b1d-4d037c55c073-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n26v8\" (UID: \"b02109e0-dfcd-4633-9b1d-4d037c55c073\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:32.183481 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.183461 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b02109e0-dfcd-4633-9b1d-4d037c55c073-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n26v8\" (UID: \"b02109e0-dfcd-4633-9b1d-4d037c55c073\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:32.198348 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.198323 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdx7\" (UniqueName: \"kubernetes.io/projected/9ff6b759-d3d3-4d1e-b216-02a81309f907-kube-api-access-qvdx7\") pod \"limitador-operator-controller-manager-859d7b78bd-jszsl\" (UID: \"9ff6b759-d3d3-4d1e-b216-02a81309f907\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" Jul 16 17:38:32.199540 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.199518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcxv\" (UniqueName: \"kubernetes.io/projected/b02109e0-dfcd-4633-9b1d-4d037c55c073-kube-api-access-nwcxv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n26v8\" (UID: \"b02109e0-dfcd-4633-9b1d-4d037c55c073\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:32.283524 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.283485 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/692d87fa-192c-4372-a34d-a5e6fe4d8b09-kube-api-access-g4jdj\") pod \"692d87fa-192c-4372-a34d-a5e6fe4d8b09\" (UID: \"692d87fa-192c-4372-a34d-a5e6fe4d8b09\") " Jul 16 17:38:32.283524 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.283524 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d4rw\" (UniqueName: \"kubernetes.io/projected/2f4f10b1-a8b5-46fb-abae-29e595045d0b-kube-api-access-4d4rw\") pod \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\" (UID: \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\") " Jul 16 17:38:32.283719 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.283575 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f4f10b1-a8b5-46fb-abae-29e595045d0b-extensions-socket-volume\") pod \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\" (UID: \"2f4f10b1-a8b5-46fb-abae-29e595045d0b\") " Jul 16 17:38:32.284128 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.284096 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4f10b1-a8b5-46fb-abae-29e595045d0b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2f4f10b1-a8b5-46fb-abae-29e595045d0b" (UID: "2f4f10b1-a8b5-46fb-abae-29e595045d0b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:38:32.285854 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.285830 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4f10b1-a8b5-46fb-abae-29e595045d0b-kube-api-access-4d4rw" (OuterVolumeSpecName: "kube-api-access-4d4rw") pod "2f4f10b1-a8b5-46fb-abae-29e595045d0b" (UID: "2f4f10b1-a8b5-46fb-abae-29e595045d0b"). InnerVolumeSpecName "kube-api-access-4d4rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:38:32.285943 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.285872 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692d87fa-192c-4372-a34d-a5e6fe4d8b09-kube-api-access-g4jdj" (OuterVolumeSpecName: "kube-api-access-g4jdj") pod "692d87fa-192c-4372-a34d-a5e6fe4d8b09" (UID: "692d87fa-192c-4372-a34d-a5e6fe4d8b09"). InnerVolumeSpecName "kube-api-access-g4jdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:38:32.294112 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.294089 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:32.299158 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.299134 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" Jul 16 17:38:32.384930 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.384901 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/692d87fa-192c-4372-a34d-a5e6fe4d8b09-kube-api-access-g4jdj\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:38:32.384930 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.384933 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4rw\" (UniqueName: \"kubernetes.io/projected/2f4f10b1-a8b5-46fb-abae-29e595045d0b-kube-api-access-4d4rw\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:38:32.385144 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.384943 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f4f10b1-a8b5-46fb-abae-29e595045d0b-extensions-socket-volume\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:38:32.443640 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.443530 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8"] Jul 16 17:38:32.446611 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:38:32.446586 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02109e0_dfcd_4633_9b1d_4d037c55c073.slice/crio-d71077a413c7d9f6f8dedbdcadae775b18cc0696418ef1d058d433f7384a355d WatchSource:0}: Error finding container d71077a413c7d9f6f8dedbdcadae775b18cc0696418ef1d058d433f7384a355d: Status 404 returned error can't find the container with id d71077a413c7d9f6f8dedbdcadae775b18cc0696418ef1d058d433f7384a355d Jul 16 17:38:32.473244 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.473223 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl"] Jul 16 17:38:32.479467 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:38:32.479440 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff6b759_d3d3_4d1e_b216_02a81309f907.slice/crio-01c00b6a73a4cb136e9aff8720a0eb96d36077ea7ef14c7be9ef3737df82f755 WatchSource:0}: Error finding container 01c00b6a73a4cb136e9aff8720a0eb96d36077ea7ef14c7be9ef3737df82f755: Status 404 returned error can't find the container with id 01c00b6a73a4cb136e9aff8720a0eb96d36077ea7ef14c7be9ef3737df82f755 Jul 16 17:38:32.651974 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.651894 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4f10b1-a8b5-46fb-abae-29e595045d0b" path="/var/lib/kubelet/pods/2f4f10b1-a8b5-46fb-abae-29e595045d0b/volumes" Jul 16 17:38:32.652245 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:32.652232 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692d87fa-192c-4372-a34d-a5e6fe4d8b09" path="/var/lib/kubelet/pods/692d87fa-192c-4372-a34d-a5e6fe4d8b09/volumes" Jul 16 17:38:33.089430 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.089404 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vjbdv" Jul 16 17:38:33.089937 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.089442 2582 scope.go:117] "RemoveContainer" containerID="c0c5805fb4afe8ec74441dfba516a7c723d76e48ef6b6ebcda2a49a9f16fc9e0" Jul 16 17:38:33.091505 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.091478 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" event={"ID":"b02109e0-dfcd-4633-9b1d-4d037c55c073","Type":"ContainerStarted","Data":"4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4"} Jul 16 17:38:33.091619 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.091536 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" event={"ID":"b02109e0-dfcd-4633-9b1d-4d037c55c073","Type":"ContainerStarted","Data":"d71077a413c7d9f6f8dedbdcadae775b18cc0696418ef1d058d433f7384a355d"} Jul 16 17:38:33.091619 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.091585 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:33.092992 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.092975 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4qpl6" Jul 16 17:38:33.094590 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.094566 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" event={"ID":"9ff6b759-d3d3-4d1e-b216-02a81309f907","Type":"ContainerStarted","Data":"bcf695719e4eaaab352ed3d7d865a4f408d2976091636505588fbc4de90b1641"} Jul 16 17:38:33.094686 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.094615 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" event={"ID":"9ff6b759-d3d3-4d1e-b216-02a81309f907","Type":"ContainerStarted","Data":"01c00b6a73a4cb136e9aff8720a0eb96d36077ea7ef14c7be9ef3737df82f755"} Jul 16 17:38:33.094754 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.094703 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" Jul 16 17:38:33.101626 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.101611 2582 scope.go:117] "RemoveContainer" containerID="6f82f51fc4eadae3fe2f2df3069e7c5947c48b0d56a9133b154abb9a39b90750" Jul 16 17:38:33.116553 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.116510 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" podStartSLOduration=2.116494874 podStartE2EDuration="2.116494874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:33.114317572 +0000 UTC m=+574.865957150" watchObservedRunningTime="2026-07-16 17:38:33.116494874 +0000 UTC m=+574.868134432" Jul 16 17:38:33.141435 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:33.137147 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" podStartSLOduration=2.137124418 podStartE2EDuration="2.137124418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:33.134380617 +0000 UTC m=+574.886020195" watchObservedRunningTime="2026-07-16 17:38:33.137124418 +0000 UTC m=+574.888763977" Jul 16 17:38:44.104313 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:44.104249 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:38:44.104775 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:38:44.104469 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jszsl" Jul 16 17:39:03.856703 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:03.856667 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8"] Jul 16 17:39:03.857137 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:03.856868 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" podUID="b02109e0-dfcd-4633-9b1d-4d037c55c073" containerName="manager" containerID="cri-o://4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4" gracePeriod=10 Jul 16 17:39:04.122771 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.122744 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:39:04.194544 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.194507 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8"] Jul 16 17:39:04.194886 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.194874 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02109e0-dfcd-4633-9b1d-4d037c55c073" containerName="manager" Jul 16 17:39:04.194933 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.194897 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02109e0-dfcd-4633-9b1d-4d037c55c073" containerName="manager" Jul 16 17:39:04.194966 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.194953 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02109e0-dfcd-4633-9b1d-4d037c55c073" containerName="manager" Jul 16 17:39:04.198054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.198038 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:04.215952 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.215920 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8"] Jul 16 17:39:04.257532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.257494 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b02109e0-dfcd-4633-9b1d-4d037c55c073-extensions-socket-volume\") pod \"b02109e0-dfcd-4633-9b1d-4d037c55c073\" (UID: \"b02109e0-dfcd-4633-9b1d-4d037c55c073\") " Jul 16 17:39:04.257825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.257581 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwcxv\" (UniqueName: \"kubernetes.io/projected/b02109e0-dfcd-4633-9b1d-4d037c55c073-kube-api-access-nwcxv\") pod \"b02109e0-dfcd-4633-9b1d-4d037c55c073\" (UID: \"b02109e0-dfcd-4633-9b1d-4d037c55c073\") " Jul 16 17:39:04.257825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.257712 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fl7g8\" (UID: \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:04.257825 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.257767 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvgg\" (UniqueName: \"kubernetes.io/projected/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-kube-api-access-2gvgg\") pod \"kuadrant-operator-controller-manager-86878c769f-fl7g8\" (UID: \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:04.257989 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.257964 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02109e0-dfcd-4633-9b1d-4d037c55c073-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b02109e0-dfcd-4633-9b1d-4d037c55c073" (UID: "b02109e0-dfcd-4633-9b1d-4d037c55c073"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:39:04.260274 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.260249 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02109e0-dfcd-4633-9b1d-4d037c55c073-kube-api-access-nwcxv" (OuterVolumeSpecName: "kube-api-access-nwcxv") pod "b02109e0-dfcd-4633-9b1d-4d037c55c073" (UID: "b02109e0-dfcd-4633-9b1d-4d037c55c073"). InnerVolumeSpecName "kube-api-access-nwcxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:39:04.263981 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.263954 2582 generic.go:356] "Generic (PLEG): container finished" podID="b02109e0-dfcd-4633-9b1d-4d037c55c073" containerID="4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4" exitCode=0 Jul 16 17:39:04.264085 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.264000 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" event={"ID":"b02109e0-dfcd-4633-9b1d-4d037c55c073","Type":"ContainerDied","Data":"4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4"} Jul 16 17:39:04.264085 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.264006 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" Jul 16 17:39:04.264085 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.264046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" event={"ID":"b02109e0-dfcd-4633-9b1d-4d037c55c073","Type":"ContainerDied","Data":"d71077a413c7d9f6f8dedbdcadae775b18cc0696418ef1d058d433f7384a355d"} Jul 16 17:39:04.264085 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.264069 2582 scope.go:117] "RemoveContainer" containerID="4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4" Jul 16 17:39:04.273159 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.273139 2582 scope.go:117] "RemoveContainer" containerID="4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4" Jul 16 17:39:04.273453 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:39:04.273433 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4\": container with ID starting with 4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4 not found: ID does not exist" containerID="4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4" Jul 16 17:39:04.273532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.273468 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4"} err="failed to get container status \"4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4\": rpc error: code = NotFound desc = could not find container \"4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4\": container with ID starting with 4298d44b9ae725d38e6a49d0144fa75f14880acf26b735ab55e1bc5dc006bed4 not found: ID does not exist" Jul 16 17:39:04.291054 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.291023 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8"] Jul 16 17:39:04.297573 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.297546 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8"] Jul 16 17:39:04.358808 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.358716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fl7g8\" (UID: \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:04.358808 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.358766 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvgg\" (UniqueName: \"kubernetes.io/projected/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-kube-api-access-2gvgg\") pod \"kuadrant-operator-controller-manager-86878c769f-fl7g8\" (UID: \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:04.359027 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.358825 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b02109e0-dfcd-4633-9b1d-4d037c55c073-extensions-socket-volume\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:39:04.359027 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.358835 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcxv\" (UniqueName: \"kubernetes.io/projected/b02109e0-dfcd-4633-9b1d-4d037c55c073-kube-api-access-nwcxv\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:39:04.359208 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.359182 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fl7g8\" (UID: \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:04.369455 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.369429 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvgg\" (UniqueName: \"kubernetes.io/projected/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-kube-api-access-2gvgg\") pod \"kuadrant-operator-controller-manager-86878c769f-fl7g8\" (UID: \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:04.507990 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.507950 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:04.652722 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.652647 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02109e0-dfcd-4633-9b1d-4d037c55c073" path="/var/lib/kubelet/pods/b02109e0-dfcd-4633-9b1d-4d037c55c073/volumes" Jul 16 17:39:04.662346 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:04.662323 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8"] Jul 16 17:39:04.664220 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:39:04.664196 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcb2caa_17e1_4e08_aa0c_e90036e6cd07.slice/crio-31072dd8d7a814d8134ea4bed5ed38016882e7928b3d26f56565a220c977617c WatchSource:0}: Error finding container 31072dd8d7a814d8134ea4bed5ed38016882e7928b3d26f56565a220c977617c: Status 404 returned error can't find the container with id 31072dd8d7a814d8134ea4bed5ed38016882e7928b3d26f56565a220c977617c Jul 16 17:39:05.102652 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:05.102608 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n26v8" podUID="b02109e0-dfcd-4633-9b1d-4d037c55c073" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.41:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 17:39:05.272708 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:05.272665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" event={"ID":"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07","Type":"ContainerStarted","Data":"1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d"} Jul 16 17:39:05.272880 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:05.272728 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" event={"ID":"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07","Type":"ContainerStarted","Data":"31072dd8d7a814d8134ea4bed5ed38016882e7928b3d26f56565a220c977617c"} Jul 16 17:39:05.272880 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:05.272781 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:05.297590 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:05.297529 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" podStartSLOduration=1.297507832 podStartE2EDuration="1.297507832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:39:05.29613788 +0000 UTC m=+607.047777438" watchObservedRunningTime="2026-07-16 17:39:05.297507832 +0000 UTC m=+607.049147391" Jul 16 17:39:16.280935 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:16.280903 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:39:20.274984 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.274901 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt"] Jul 16 17:39:20.278748 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.278721 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.281638 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.281615 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xnwzs\"" Jul 16 17:39:20.291906 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.291877 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt"] Jul 16 17:39:20.398644 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.398644 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cedcd98b-0520-456c-a82b-e1c9948e8b76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.398866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398676 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.398866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.398866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.398866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.398866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqxx\" (UniqueName: \"kubernetes.io/projected/cedcd98b-0520-456c-a82b-e1c9948e8b76-kube-api-access-fvqxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.398866 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.399045 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.398901 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500219 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500219 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500220 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cedcd98b-0520-456c-a82b-e1c9948e8b76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqxx\" (UniqueName: \"kubernetes.io/projected/cedcd98b-0520-456c-a82b-e1c9948e8b76-kube-api-access-fvqxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500811 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500811 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500702 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500923 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.500923 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.501017 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.500949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.501049 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.501016 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cedcd98b-0520-456c-a82b-e1c9948e8b76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.502675 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.502657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.503000 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.502982 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.510620 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.510593 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cedcd98b-0520-456c-a82b-e1c9948e8b76-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.510813 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.510796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqxx\" (UniqueName: \"kubernetes.io/projected/cedcd98b-0520-456c-a82b-e1c9948e8b76-kube-api-access-fvqxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwrxt\" (UID: \"cedcd98b-0520-456c-a82b-e1c9948e8b76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.588989 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.588936 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:20.762460 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.762431 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt"] Jul 16 17:39:20.764405 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:39:20.764380 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedcd98b_0520_456c_a82b_e1c9948e8b76.slice/crio-6b49b798f661a29c11b6f1b21ed58c2836878429ad002a8634d8278ce3d03d6f WatchSource:0}: Error finding container 6b49b798f661a29c11b6f1b21ed58c2836878429ad002a8634d8278ce3d03d6f: Status 404 returned error can't find the container with id 6b49b798f661a29c11b6f1b21ed58c2836878429ad002a8634d8278ce3d03d6f Jul 16 17:39:20.766471 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.766442 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:39:20.766544 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.766507 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:39:20.766544 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:20.766537 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:39:21.364003 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:21.363963 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" event={"ID":"cedcd98b-0520-456c-a82b-e1c9948e8b76","Type":"ContainerStarted","Data":"5f5dd8cff49dbd2895563edd573ee081ae0ce7b5d66136d7e36c7fbcbb51e464"} Jul 16 17:39:21.364412 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:21.364021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" event={"ID":"cedcd98b-0520-456c-a82b-e1c9948e8b76","Type":"ContainerStarted","Data":"6b49b798f661a29c11b6f1b21ed58c2836878429ad002a8634d8278ce3d03d6f"} Jul 16 17:39:21.389170 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:21.389116 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" podStartSLOduration=1.389098022 podStartE2EDuration="1.389098022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:39:21.388392427 +0000 UTC m=+623.140032000" watchObservedRunningTime="2026-07-16 17:39:21.389098022 +0000 UTC m=+623.140737657" Jul 16 17:39:21.589470 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:21.589435 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:21.594139 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:21.594113 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:22.369173 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:22.369143 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:22.370332 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:22.370313 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:39:23.818698 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:23.818661 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wh4rj"] Jul 16 17:39:23.822230 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:23.822205 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wh4rj" Jul 16 17:39:23.824913 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:23.824878 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-djg2f\"" Jul 16 17:39:23.827829 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:23.827803 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wh4rj"] Jul 16 17:39:23.933908 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:23.933874 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vmd\" (UniqueName: \"kubernetes.io/projected/618897e1-2daa-4230-8f47-8e6b9f2ea18f-kube-api-access-m7vmd\") pod \"authorino-58cd955675-wh4rj\" (UID: \"618897e1-2daa-4230-8f47-8e6b9f2ea18f\") " pod="kuadrant-system/authorino-58cd955675-wh4rj" Jul 16 17:39:24.034430 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:24.034391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vmd\" (UniqueName: \"kubernetes.io/projected/618897e1-2daa-4230-8f47-8e6b9f2ea18f-kube-api-access-m7vmd\") pod \"authorino-58cd955675-wh4rj\" (UID: \"618897e1-2daa-4230-8f47-8e6b9f2ea18f\") " pod="kuadrant-system/authorino-58cd955675-wh4rj" Jul 16 17:39:24.044510 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:24.044484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vmd\" (UniqueName: \"kubernetes.io/projected/618897e1-2daa-4230-8f47-8e6b9f2ea18f-kube-api-access-m7vmd\") pod \"authorino-58cd955675-wh4rj\" (UID: \"618897e1-2daa-4230-8f47-8e6b9f2ea18f\") " pod="kuadrant-system/authorino-58cd955675-wh4rj" Jul 16 17:39:24.133833 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:24.133746 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wh4rj" Jul 16 17:39:24.280978 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:24.280950 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wh4rj"] Jul 16 17:39:24.282361 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:39:24.282331 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod618897e1_2daa_4230_8f47_8e6b9f2ea18f.slice/crio-7973bdf621a96670450e7b7553ee4862ec2686bb96108e925d8b6aca3947fda0 WatchSource:0}: Error finding container 7973bdf621a96670450e7b7553ee4862ec2686bb96108e925d8b6aca3947fda0: Status 404 returned error can't find the container with id 7973bdf621a96670450e7b7553ee4862ec2686bb96108e925d8b6aca3947fda0 Jul 16 17:39:24.380306 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:24.380254 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wh4rj" event={"ID":"618897e1-2daa-4230-8f47-8e6b9f2ea18f","Type":"ContainerStarted","Data":"7973bdf621a96670450e7b7553ee4862ec2686bb96108e925d8b6aca3947fda0"} Jul 16 17:39:28.406344 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:28.406303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wh4rj" event={"ID":"618897e1-2daa-4230-8f47-8e6b9f2ea18f","Type":"ContainerStarted","Data":"90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550"} Jul 16 17:39:28.424154 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:28.424070 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wh4rj" podStartSLOduration=2.135893509 podStartE2EDuration="5.424051005s" totalImagesPullingTime="3.288157496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:39:28.423126671 +0000 UTC m=+630.174766229" watchObservedRunningTime="2026-07-16 17:39:28.424051005 +0000 UTC m=+630.175690562" Jul 16 17:39:29.041686 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:29.041645 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wh4rj"] Jul 16 17:39:30.416311 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:30.416258 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wh4rj" podUID="618897e1-2daa-4230-8f47-8e6b9f2ea18f" containerName="authorino" containerID="cri-o://90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550" gracePeriod=30 Jul 16 17:39:30.665807 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:30.665773 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wh4rj" Jul 16 17:39:30.686247 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:30.686163 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7vmd\" (UniqueName: \"kubernetes.io/projected/618897e1-2daa-4230-8f47-8e6b9f2ea18f-kube-api-access-m7vmd\") pod \"618897e1-2daa-4230-8f47-8e6b9f2ea18f\" (UID: \"618897e1-2daa-4230-8f47-8e6b9f2ea18f\") " Jul 16 17:39:30.688748 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:30.688705 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618897e1-2daa-4230-8f47-8e6b9f2ea18f-kube-api-access-m7vmd" (OuterVolumeSpecName: "kube-api-access-m7vmd") pod "618897e1-2daa-4230-8f47-8e6b9f2ea18f" (UID: "618897e1-2daa-4230-8f47-8e6b9f2ea18f"). InnerVolumeSpecName "kube-api-access-m7vmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:39:30.786917 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:30.786866 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vmd\" (UniqueName: \"kubernetes.io/projected/618897e1-2daa-4230-8f47-8e6b9f2ea18f-kube-api-access-m7vmd\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:39:31.423609 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.423572 2582 generic.go:356] "Generic (PLEG): container finished" podID="618897e1-2daa-4230-8f47-8e6b9f2ea18f" containerID="90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550" exitCode=0 Jul 16 17:39:31.424058 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.423616 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wh4rj" Jul 16 17:39:31.424058 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.423657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wh4rj" event={"ID":"618897e1-2daa-4230-8f47-8e6b9f2ea18f","Type":"ContainerDied","Data":"90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550"} Jul 16 17:39:31.424058 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.423698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wh4rj" event={"ID":"618897e1-2daa-4230-8f47-8e6b9f2ea18f","Type":"ContainerDied","Data":"7973bdf621a96670450e7b7553ee4862ec2686bb96108e925d8b6aca3947fda0"} Jul 16 17:39:31.424058 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.423716 2582 scope.go:117] "RemoveContainer" containerID="90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550" Jul 16 17:39:31.433610 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.433591 2582 scope.go:117] "RemoveContainer" containerID="90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550" Jul 16 17:39:31.433955 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:39:31.433936 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550\": container with ID starting with 90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550 not found: ID does not exist" containerID="90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550" Jul 16 17:39:31.434023 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.433966 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550"} err="failed to get container status \"90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550\": rpc error: code = NotFound desc = could not find container \"90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550\": container with ID starting with 90aab6f864a8a5bf6984bdba790beb055ea8e399211aff5ade3c13fa9d91e550 not found: ID does not exist" Jul 16 17:39:31.448299 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.448250 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wh4rj"] Jul 16 17:39:31.452845 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:31.452817 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wh4rj"] Jul 16 17:39:32.651980 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:39:32.651946 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618897e1-2daa-4230-8f47-8e6b9f2ea18f" path="/var/lib/kubelet/pods/618897e1-2daa-4230-8f47-8e6b9f2ea18f/volumes" Jul 16 17:40:04.191078 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.191021 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bfdf69b79-pddqm"] Jul 16 17:40:04.191603 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.191418 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618897e1-2daa-4230-8f47-8e6b9f2ea18f" containerName="authorino" Jul 16 17:40:04.191603 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.191431 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="618897e1-2daa-4230-8f47-8e6b9f2ea18f" containerName="authorino" Jul 16 17:40:04.191603 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.191492 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="618897e1-2daa-4230-8f47-8e6b9f2ea18f" containerName="authorino" Jul 16 17:40:04.195893 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.195873 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:04.200204 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.200179 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 17:40:04.200374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.200227 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 16 17:40:04.200374 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.200235 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9vnw5\"" Jul 16 17:40:04.204836 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.204809 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bfdf69b79-pddqm"] Jul 16 17:40:04.290037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.290002 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqt2\" (UniqueName: \"kubernetes.io/projected/2e1bee41-64e6-4dca-80a8-76253acdca81-kube-api-access-4rqt2\") pod \"maas-controller-bfdf69b79-pddqm\" (UID: \"2e1bee41-64e6-4dca-80a8-76253acdca81\") " pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:04.290220 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.290079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e1bee41-64e6-4dca-80a8-76253acdca81-webhook-cert\") pod \"maas-controller-bfdf69b79-pddqm\" (UID: \"2e1bee41-64e6-4dca-80a8-76253acdca81\") " pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:04.390832 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.390796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqt2\" (UniqueName: \"kubernetes.io/projected/2e1bee41-64e6-4dca-80a8-76253acdca81-kube-api-access-4rqt2\") pod \"maas-controller-bfdf69b79-pddqm\" (UID: \"2e1bee41-64e6-4dca-80a8-76253acdca81\") " pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:04.391011 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.390869 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e1bee41-64e6-4dca-80a8-76253acdca81-webhook-cert\") pod \"maas-controller-bfdf69b79-pddqm\" (UID: \"2e1bee41-64e6-4dca-80a8-76253acdca81\") " pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:04.393679 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.393645 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e1bee41-64e6-4dca-80a8-76253acdca81-webhook-cert\") pod \"maas-controller-bfdf69b79-pddqm\" (UID: \"2e1bee41-64e6-4dca-80a8-76253acdca81\") " pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:04.401508 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.401479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqt2\" (UniqueName: \"kubernetes.io/projected/2e1bee41-64e6-4dca-80a8-76253acdca81-kube-api-access-4rqt2\") pod \"maas-controller-bfdf69b79-pddqm\" (UID: \"2e1bee41-64e6-4dca-80a8-76253acdca81\") " pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:04.508696 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.508604 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:04.652193 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:04.652160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bfdf69b79-pddqm"] Jul 16 17:40:04.653593 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:40:04.653566 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1bee41_64e6_4dca_80a8_76253acdca81.slice/crio-473fcd265b9521e486e1d43643a1b171bc51c8eca97d8dd96d5e690b9a490747 WatchSource:0}: Error finding container 473fcd265b9521e486e1d43643a1b171bc51c8eca97d8dd96d5e690b9a490747: Status 404 returned error can't find the container with id 473fcd265b9521e486e1d43643a1b171bc51c8eca97d8dd96d5e690b9a490747 Jul 16 17:40:05.607125 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:05.607080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bfdf69b79-pddqm" event={"ID":"2e1bee41-64e6-4dca-80a8-76253acdca81","Type":"ContainerStarted","Data":"473fcd265b9521e486e1d43643a1b171bc51c8eca97d8dd96d5e690b9a490747"} Jul 16 17:40:07.619653 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:07.619606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bfdf69b79-pddqm" event={"ID":"2e1bee41-64e6-4dca-80a8-76253acdca81","Type":"ContainerStarted","Data":"fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a"} Jul 16 17:40:07.620131 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:07.619730 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:07.649888 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:07.649780 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bfdf69b79-pddqm" podStartSLOduration=0.912631518 podStartE2EDuration="3.649763993s" totalImagesPullingTime="2.737132475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:40:07.647784384 +0000 UTC m=+669.399423957" watchObservedRunningTime="2026-07-16 17:40:07.649763993 +0000 UTC m=+669.401403603" Jul 16 17:40:18.629466 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:18.629427 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:19.278425 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.278387 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bfdf69b79-pddqm"] Jul 16 17:40:19.278618 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.278586 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bfdf69b79-pddqm" podUID="2e1bee41-64e6-4dca-80a8-76253acdca81" containerName="manager" containerID="cri-o://fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a" gracePeriod=30 Jul 16 17:40:19.529145 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.529075 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:19.619553 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.619522 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rqt2\" (UniqueName: \"kubernetes.io/projected/2e1bee41-64e6-4dca-80a8-76253acdca81-kube-api-access-4rqt2\") pod \"2e1bee41-64e6-4dca-80a8-76253acdca81\" (UID: \"2e1bee41-64e6-4dca-80a8-76253acdca81\") " Jul 16 17:40:19.619726 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.619565 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e1bee41-64e6-4dca-80a8-76253acdca81-webhook-cert\") pod \"2e1bee41-64e6-4dca-80a8-76253acdca81\" (UID: \"2e1bee41-64e6-4dca-80a8-76253acdca81\") " Jul 16 17:40:19.621971 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.621936 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1bee41-64e6-4dca-80a8-76253acdca81-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2e1bee41-64e6-4dca-80a8-76253acdca81" (UID: "2e1bee41-64e6-4dca-80a8-76253acdca81"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:40:19.622074 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.621999 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1bee41-64e6-4dca-80a8-76253acdca81-kube-api-access-4rqt2" (OuterVolumeSpecName: "kube-api-access-4rqt2") pod "2e1bee41-64e6-4dca-80a8-76253acdca81" (UID: "2e1bee41-64e6-4dca-80a8-76253acdca81"). InnerVolumeSpecName "kube-api-access-4rqt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:40:19.683235 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.683194 2582 generic.go:356] "Generic (PLEG): container finished" podID="2e1bee41-64e6-4dca-80a8-76253acdca81" containerID="fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a" exitCode=0 Jul 16 17:40:19.683612 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.683241 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bfdf69b79-pddqm" Jul 16 17:40:19.683612 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.683294 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bfdf69b79-pddqm" event={"ID":"2e1bee41-64e6-4dca-80a8-76253acdca81","Type":"ContainerDied","Data":"fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a"} Jul 16 17:40:19.683612 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.683353 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bfdf69b79-pddqm" event={"ID":"2e1bee41-64e6-4dca-80a8-76253acdca81","Type":"ContainerDied","Data":"473fcd265b9521e486e1d43643a1b171bc51c8eca97d8dd96d5e690b9a490747"} Jul 16 17:40:19.683612 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.683379 2582 scope.go:117] "RemoveContainer" containerID="fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a" Jul 16 17:40:19.692752 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.692732 2582 scope.go:117] "RemoveContainer" containerID="fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a" Jul 16 17:40:19.693045 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:40:19.693026 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a\": container with ID starting with fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a not found: ID does not exist" containerID="fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a" Jul 16 17:40:19.693105 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.693055 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a"} err="failed to get container status \"fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a\": rpc error: code = NotFound desc = could not find container \"fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a\": container with ID starting with fec13048dd51bf5f0711a9c31a65b763d002449ccdac719581927dad4ba2972a not found: ID does not exist" Jul 16 17:40:19.718386 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.718351 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bfdf69b79-pddqm"] Jul 16 17:40:19.720768 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.720732 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqt2\" (UniqueName: \"kubernetes.io/projected/2e1bee41-64e6-4dca-80a8-76253acdca81-kube-api-access-4rqt2\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:40:19.720768 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.720759 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e1bee41-64e6-4dca-80a8-76253acdca81-webhook-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:40:19.724319 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:19.724295 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bfdf69b79-pddqm"] Jul 16 17:40:20.651634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:20.651599 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1bee41-64e6-4dca-80a8-76253acdca81" path="/var/lib/kubelet/pods/2e1bee41-64e6-4dca-80a8-76253acdca81/volumes" Jul 16 17:40:38.946928 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.946891 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4"] Jul 16 17:40:38.947354 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.947230 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1bee41-64e6-4dca-80a8-76253acdca81" containerName="manager" Jul 16 17:40:38.947354 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.947240 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1bee41-64e6-4dca-80a8-76253acdca81" containerName="manager" Jul 16 17:40:38.947354 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.947348 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1bee41-64e6-4dca-80a8-76253acdca81" containerName="manager" Jul 16 17:40:38.950546 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.950526 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:38.953395 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.953373 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 16 17:40:38.953519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.953400 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:40:38.953519 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.953476 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:40:38.959939 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.959915 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-openshift-service-ca\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:38.960025 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.959952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66v8x\" (UniqueName: \"kubernetes.io/projected/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-kube-api-access-66v8x\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:38.960025 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.959974 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-maas-api-tls\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:38.966424 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:38.966401 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4"] Jul 16 17:40:39.060818 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.060784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66v8x\" (UniqueName: \"kubernetes.io/projected/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-kube-api-access-66v8x\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:39.060818 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.060824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-maas-api-tls\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:39.061034 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.060890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-openshift-service-ca\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:39.061034 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:40:39.060993 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 16 17:40:39.061099 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:40:39.061070 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-maas-api-tls podName:96b6891f-ae9d-416e-bcbe-9f08edfee3cb nodeName:}" failed. No retries permitted until 2026-07-16 17:40:39.561053973 +0000 UTC m=+701.312693509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-maas-api-tls") pod "maas-api-d67b469d5-pwdk4" (UID: "96b6891f-ae9d-416e-bcbe-9f08edfee3cb") : secret "maas-api-serving-cert" not found Jul 16 17:40:39.061509 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.061491 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-openshift-service-ca\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:39.075789 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.075755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66v8x\" (UniqueName: \"kubernetes.io/projected/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-kube-api-access-66v8x\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:39.565686 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.565650 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-maas-api-tls\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:39.568163 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.568136 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b6891f-ae9d-416e-bcbe-9f08edfee3cb-maas-api-tls\") pod \"maas-api-d67b469d5-pwdk4\" (UID: \"96b6891f-ae9d-416e-bcbe-9f08edfee3cb\") " pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:39.752525 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.752491 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-8x4ks"] Jul 16 17:40:39.756188 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.756152 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:39.759533 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.759504 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 17:40:39.759822 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.759793 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-srwcz\"" Jul 16 17:40:39.767039 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.767014 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-8x4ks"] Jul 16 17:40:39.767146 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.767054 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003f7852-ea9b-4158-926d-303d18d7d8fc-plugins-config-volume\") pod \"payload-processing-5bf47cd988-8x4ks\" (UID: \"003f7852-ea9b-4158-926d-303d18d7d8fc\") " pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:39.767196 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.767147 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nj5s\" (UniqueName: \"kubernetes.io/projected/003f7852-ea9b-4158-926d-303d18d7d8fc-kube-api-access-6nj5s\") pod \"payload-processing-5bf47cd988-8x4ks\" (UID: \"003f7852-ea9b-4158-926d-303d18d7d8fc\") " pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:39.860932 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.860840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:39.868127 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.868098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003f7852-ea9b-4158-926d-303d18d7d8fc-plugins-config-volume\") pod \"payload-processing-5bf47cd988-8x4ks\" (UID: \"003f7852-ea9b-4158-926d-303d18d7d8fc\") " pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:39.868267 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.868186 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nj5s\" (UniqueName: \"kubernetes.io/projected/003f7852-ea9b-4158-926d-303d18d7d8fc-kube-api-access-6nj5s\") pod \"payload-processing-5bf47cd988-8x4ks\" (UID: \"003f7852-ea9b-4158-926d-303d18d7d8fc\") " pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:39.868903 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.868882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003f7852-ea9b-4158-926d-303d18d7d8fc-plugins-config-volume\") pod \"payload-processing-5bf47cd988-8x4ks\" (UID: \"003f7852-ea9b-4158-926d-303d18d7d8fc\") " pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:39.880719 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:39.880689 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nj5s\" (UniqueName: \"kubernetes.io/projected/003f7852-ea9b-4158-926d-303d18d7d8fc-kube-api-access-6nj5s\") pod \"payload-processing-5bf47cd988-8x4ks\" (UID: \"003f7852-ea9b-4158-926d-303d18d7d8fc\") " pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:40.028653 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:40.028630 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4"] Jul 16 17:40:40.029802 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:40:40.029778 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b6891f_ae9d_416e_bcbe_9f08edfee3cb.slice/crio-9c73f14b964f67a991fc99226b475c8e39c5e5fdb2ef367ded3fac1dce4dc979 WatchSource:0}: Error finding container 9c73f14b964f67a991fc99226b475c8e39c5e5fdb2ef367ded3fac1dce4dc979: Status 404 returned error can't find the container with id 9c73f14b964f67a991fc99226b475c8e39c5e5fdb2ef367ded3fac1dce4dc979 Jul 16 17:40:40.031121 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:40.031103 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:40:40.067726 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:40.067691 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:40.222251 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:40.222222 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-8x4ks"] Jul 16 17:40:40.223480 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:40:40.223451 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003f7852_ea9b_4158_926d_303d18d7d8fc.slice/crio-4ff63766fd6e183471a6ca060684dead50f65b7f99a12649928403ba36b1093e WatchSource:0}: Error finding container 4ff63766fd6e183471a6ca060684dead50f65b7f99a12649928403ba36b1093e: Status 404 returned error can't find the container with id 4ff63766fd6e183471a6ca060684dead50f65b7f99a12649928403ba36b1093e Jul 16 17:40:40.815924 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:40.815857 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" event={"ID":"96b6891f-ae9d-416e-bcbe-9f08edfee3cb","Type":"ContainerStarted","Data":"9c73f14b964f67a991fc99226b475c8e39c5e5fdb2ef367ded3fac1dce4dc979"} Jul 16 17:40:40.818041 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:40.818001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" event={"ID":"003f7852-ea9b-4158-926d-303d18d7d8fc","Type":"ContainerStarted","Data":"4ff63766fd6e183471a6ca060684dead50f65b7f99a12649928403ba36b1093e"} Jul 16 17:40:42.831259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:42.831214 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" event={"ID":"96b6891f-ae9d-416e-bcbe-9f08edfee3cb","Type":"ContainerStarted","Data":"7d0eeacb25ff01541060f1137cc63d28549f8b9ef22a286093b589b6e91a8d6a"} Jul 16 17:40:42.831727 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:42.831323 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:42.832860 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:42.832834 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" event={"ID":"003f7852-ea9b-4158-926d-303d18d7d8fc","Type":"ContainerStarted","Data":"6a02e5bb82fb5c5831f281f1ec64edecc41219b6e949481620c226b9586792b1"} Jul 16 17:40:42.833000 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:42.832901 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:40:42.853236 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:42.853184 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" podStartSLOduration=2.536667499 podStartE2EDuration="4.853167476s" totalImagesPullingTime="2.316499977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:40:42.852073139 +0000 UTC m=+704.603712718" watchObservedRunningTime="2026-07-16 17:40:42.853167476 +0000 UTC m=+704.604807034" Jul 16 17:40:42.871829 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:42.871770 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" podStartSLOduration=1.741538611 podStartE2EDuration="3.871751876s" totalImagesPullingTime="2.130213265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:40:42.8691164 +0000 UTC m=+704.620755957" watchObservedRunningTime="2026-07-16 17:40:42.871751876 +0000 UTC m=+704.623391466" Jul 16 17:40:47.861730 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:47.861697 2582 generic.go:356] "Generic (PLEG): container finished" podID="cedcd98b-0520-456c-a82b-e1c9948e8b76" containerID="5f5dd8cff49dbd2895563edd573ee081ae0ce7b5d66136d7e36c7fbcbb51e464" exitCode=0 Jul 16 17:40:47.862134 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:47.861767 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" event={"ID":"cedcd98b-0520-456c-a82b-e1c9948e8b76","Type":"ContainerDied","Data":"5f5dd8cff49dbd2895563edd573ee081ae0ce7b5d66136d7e36c7fbcbb51e464"} Jul 16 17:40:47.862175 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:47.862137 2582 scope.go:117] "RemoveContainer" containerID="5f5dd8cff49dbd2895563edd573ee081ae0ce7b5d66136d7e36c7fbcbb51e464" Jul 16 17:40:47.863131 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:47.863103 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:40:47.863249 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:47.863167 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:40:47.863249 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:47.863199 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:40:48.844548 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:48.844516 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d67b469d5-pwdk4" Jul 16 17:40:48.869222 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:48.869183 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" event={"ID":"cedcd98b-0520-456c-a82b-e1c9948e8b76","Type":"ContainerStarted","Data":"96039242dc97d08667595bac3077d692fed88a542aaa7b4d56b81ce0aa5d72a7"} Jul 16 17:40:49.591683 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:49.591587 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:40:49.591889 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:49.591771 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bwrxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 16 17:40:49.591889 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:49.591848 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" podUID="cedcd98b-0520-456c-a82b-e1c9948e8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 16 17:40:50.590082 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:50.590043 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bwrxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 16 17:40:50.591090 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:50.590660 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" podUID="cedcd98b-0520-456c-a82b-e1c9948e8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 16 17:40:51.589659 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:51.589607 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bwrxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 16 17:40:51.589875 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:51.589684 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" podUID="cedcd98b-0520-456c-a82b-e1c9948e8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 16 17:40:52.369245 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:52.369210 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:40:52.589839 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:52.589789 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bwrxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 16 17:40:52.590050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:52.589866 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" podUID="cedcd98b-0520-456c-a82b-e1c9948e8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 16 17:40:53.593743 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:53.593714 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:40:53.594835 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:53.594815 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwrxt" Jul 16 17:40:53.841253 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:40:53.841226 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-8x4ks" Jul 16 17:41:33.176083 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.176047 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54"] Jul 16 17:41:33.179757 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.179733 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.188457 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.188434 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 17:41:33.189807 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.189787 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 17:41:33.189930 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.189853 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jzc82\"" Jul 16 17:41:33.189930 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.189871 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 16 17:41:33.204493 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.204461 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54"] Jul 16 17:41:33.268216 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.268181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.268216 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.268216 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45978a28-d484-4a91-aa43-0b602e5683bd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.268457 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.268241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fm6\" (UniqueName: \"kubernetes.io/projected/45978a28-d484-4a91-aa43-0b602e5683bd-kube-api-access-87fm6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.268457 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.268352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.268457 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.268398 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.268551 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.268478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.368920 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.368882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.369107 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.368932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.369107 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.368962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.369107 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.369010 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.369235 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.369132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45978a28-d484-4a91-aa43-0b602e5683bd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.369235 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.369189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fm6\" (UniqueName: \"kubernetes.io/projected/45978a28-d484-4a91-aa43-0b602e5683bd-kube-api-access-87fm6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.369396 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.369367 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.369508 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.369414 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.369508 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.369458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.371436 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.371418 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45978a28-d484-4a91-aa43-0b602e5683bd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.371679 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.371664 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45978a28-d484-4a91-aa43-0b602e5683bd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.391910 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.391876 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fm6\" (UniqueName: \"kubernetes.io/projected/45978a28-d484-4a91-aa43-0b602e5683bd-kube-api-access-87fm6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54\" (UID: \"45978a28-d484-4a91-aa43-0b602e5683bd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.490069 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.489979 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:33.647316 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:33.647293 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54"] Jul 16 17:41:33.648608 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:41:33.648585 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45978a28_d484_4a91_aa43_0b602e5683bd.slice/crio-f9609b4df391537ba538d6655e91f2b6151c6445087a5c26a1af5b37f4a27148 WatchSource:0}: Error finding container f9609b4df391537ba538d6655e91f2b6151c6445087a5c26a1af5b37f4a27148: Status 404 returned error can't find the container with id f9609b4df391537ba538d6655e91f2b6151c6445087a5c26a1af5b37f4a27148 Jul 16 17:41:34.133650 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:34.133611 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" event={"ID":"45978a28-d484-4a91-aa43-0b602e5683bd","Type":"ContainerStarted","Data":"f9609b4df391537ba538d6655e91f2b6151c6445087a5c26a1af5b37f4a27148"} Jul 16 17:41:35.141407 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:35.141370 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" event={"ID":"45978a28-d484-4a91-aa43-0b602e5683bd","Type":"ContainerStarted","Data":"8c1ad8af25278077478a95675511689170c36ea0ba00cb957a3a94174719946a"} Jul 16 17:41:35.178323 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:35.178111 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" podStartSLOduration=1.143192289 podStartE2EDuration="2.178089488s" totalImagesPullingTime="1.034897199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:35.177854584 +0000 UTC m=+756.929494161" watchObservedRunningTime="2026-07-16 17:41:35.178089488 +0000 UTC m=+756.929729046" Jul 16 17:41:43.490369 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:43.490326 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:43.490756 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:43.490408 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:43.502565 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:43.502536 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:41:44.201361 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:41:44.201331 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54" Jul 16 17:42:08.051246 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.051199 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp"] Jul 16 17:42:08.055005 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.054978 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.058204 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.058181 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 17:42:08.067260 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.067230 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp"] Jul 16 17:42:08.183689 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.183639 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.183689 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.183691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.183962 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.183819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.183962 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.183878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rctd\" (UniqueName: \"kubernetes.io/projected/a71f03ac-8e7d-44e5-904d-80b9785eeda3-kube-api-access-9rctd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.183962 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.183912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71f03ac-8e7d-44e5-904d-80b9785eeda3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.183962 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.183959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.284858 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.284824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.285050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.284881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.285050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.284900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.285050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.284935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.285050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.284963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rctd\" (UniqueName: \"kubernetes.io/projected/a71f03ac-8e7d-44e5-904d-80b9785eeda3-kube-api-access-9rctd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.285050 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.284994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71f03ac-8e7d-44e5-904d-80b9785eeda3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.285322 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.285231 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.285391 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.285380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.285468 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.285449 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.287409 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.287381 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71f03ac-8e7d-44e5-904d-80b9785eeda3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.287633 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.287616 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71f03ac-8e7d-44e5-904d-80b9785eeda3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.298673 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.298651 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rctd\" (UniqueName: \"kubernetes.io/projected/a71f03ac-8e7d-44e5-904d-80b9785eeda3-kube-api-access-9rctd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp\" (UID: \"a71f03ac-8e7d-44e5-904d-80b9785eeda3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.366267 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.366181 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:08.555628 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:08.555604 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp"] Jul 16 17:42:08.558151 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:42:08.558119 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda71f03ac_8e7d_44e5_904d_80b9785eeda3.slice/crio-f2465c44b06fe7a03e66a71202669b49adeb4636bb81270ee51e4a02f64a5b3e WatchSource:0}: Error finding container f2465c44b06fe7a03e66a71202669b49adeb4636bb81270ee51e4a02f64a5b3e: Status 404 returned error can't find the container with id f2465c44b06fe7a03e66a71202669b49adeb4636bb81270ee51e4a02f64a5b3e Jul 16 17:42:09.320204 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:09.320162 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" event={"ID":"a71f03ac-8e7d-44e5-904d-80b9785eeda3","Type":"ContainerStarted","Data":"eaf04896bfb25093c6286bdf2a80c14736dc106201ef29d15f0804c1b67215d2"} Jul 16 17:42:09.320204 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:09.320208 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" event={"ID":"a71f03ac-8e7d-44e5-904d-80b9785eeda3","Type":"ContainerStarted","Data":"f2465c44b06fe7a03e66a71202669b49adeb4636bb81270ee51e4a02f64a5b3e"} Jul 16 17:42:09.342295 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:09.342235 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" podStartSLOduration=1.160364044 podStartE2EDuration="1.342217733s" totalImagesPullingTime="181.853689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:42:09.341233198 +0000 UTC m=+791.092872780" watchObservedRunningTime="2026-07-16 17:42:09.342217733 +0000 UTC m=+791.093857290" Jul 16 17:42:18.366786 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:18.366756 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:18.367250 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:18.366803 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:18.379198 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:18.379171 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:42:19.385559 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:42:19.385530 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp" Jul 16 17:43:17.157729 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.157695 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84655594dc-wfpfq"] Jul 16 17:43:17.161151 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.161133 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84655594dc-wfpfq" Jul 16 17:43:17.170379 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.170358 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-djg2f\"" Jul 16 17:43:17.171521 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.171507 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 17:43:17.183410 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.183382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84655594dc-wfpfq"] Jul 16 17:43:17.195801 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.195766 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qrr\" (UniqueName: \"kubernetes.io/projected/7cbc6f15-9138-4ec3-8091-9ffbd626b6d0-kube-api-access-p5qrr\") pod \"authorino-84655594dc-wfpfq\" (UID: \"7cbc6f15-9138-4ec3-8091-9ffbd626b6d0\") " pod="kuadrant-system/authorino-84655594dc-wfpfq" Jul 16 17:43:17.195974 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.195944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7cbc6f15-9138-4ec3-8091-9ffbd626b6d0-tls-cert\") pod \"authorino-84655594dc-wfpfq\" (UID: \"7cbc6f15-9138-4ec3-8091-9ffbd626b6d0\") " pod="kuadrant-system/authorino-84655594dc-wfpfq" Jul 16 17:43:17.296618 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.296580 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qrr\" (UniqueName: \"kubernetes.io/projected/7cbc6f15-9138-4ec3-8091-9ffbd626b6d0-kube-api-access-p5qrr\") pod \"authorino-84655594dc-wfpfq\" (UID: \"7cbc6f15-9138-4ec3-8091-9ffbd626b6d0\") " pod="kuadrant-system/authorino-84655594dc-wfpfq" Jul 16 17:43:17.296806 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.296672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7cbc6f15-9138-4ec3-8091-9ffbd626b6d0-tls-cert\") pod \"authorino-84655594dc-wfpfq\" (UID: \"7cbc6f15-9138-4ec3-8091-9ffbd626b6d0\") " pod="kuadrant-system/authorino-84655594dc-wfpfq" Jul 16 17:43:17.299418 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.299393 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7cbc6f15-9138-4ec3-8091-9ffbd626b6d0-tls-cert\") pod \"authorino-84655594dc-wfpfq\" (UID: \"7cbc6f15-9138-4ec3-8091-9ffbd626b6d0\") " pod="kuadrant-system/authorino-84655594dc-wfpfq" Jul 16 17:43:17.312643 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.312613 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qrr\" (UniqueName: \"kubernetes.io/projected/7cbc6f15-9138-4ec3-8091-9ffbd626b6d0-kube-api-access-p5qrr\") pod \"authorino-84655594dc-wfpfq\" (UID: \"7cbc6f15-9138-4ec3-8091-9ffbd626b6d0\") " pod="kuadrant-system/authorino-84655594dc-wfpfq" Jul 16 17:43:17.470601 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.470506 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84655594dc-wfpfq" Jul 16 17:43:17.647375 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.647343 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84655594dc-wfpfq"] Jul 16 17:43:17.648237 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:43:17.648210 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cbc6f15_9138_4ec3_8091_9ffbd626b6d0.slice/crio-d90770c602014ccc3cf6a0dd96d704cb7bdaa859e98e68a2b7e682ba337fc8de WatchSource:0}: Error finding container d90770c602014ccc3cf6a0dd96d704cb7bdaa859e98e68a2b7e682ba337fc8de: Status 404 returned error can't find the container with id d90770c602014ccc3cf6a0dd96d704cb7bdaa859e98e68a2b7e682ba337fc8de Jul 16 17:43:17.694231 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:17.694197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84655594dc-wfpfq" event={"ID":"7cbc6f15-9138-4ec3-8091-9ffbd626b6d0","Type":"ContainerStarted","Data":"d90770c602014ccc3cf6a0dd96d704cb7bdaa859e98e68a2b7e682ba337fc8de"} Jul 16 17:43:18.701638 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:18.701594 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84655594dc-wfpfq" event={"ID":"7cbc6f15-9138-4ec3-8091-9ffbd626b6d0","Type":"ContainerStarted","Data":"2e9cdad4c24228ff1baa997abe7ac4a130d273386a6c88c914b3a874bafdf8ec"} Jul 16 17:43:18.735158 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:43:18.735101 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84655594dc-wfpfq" podStartSLOduration=1.278084362 podStartE2EDuration="1.735083473s" totalImagesPullingTime="456.999111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:18.731649067 +0000 UTC m=+860.483288626" watchObservedRunningTime="2026-07-16 17:43:18.735083473 +0000 UTC m=+860.486723031" Jul 16 17:50:20.276568 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:20.276475 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8"] Jul 16 17:50:20.277135 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:20.276684 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" podUID="cbcb2caa-17e1-4e08-aa0c-e90036e6cd07" containerName="manager" containerID="cri-o://1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d" gracePeriod=10 Jul 16 17:50:24.243259 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:24.243230 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:50:24.357387 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:24.357300 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gvgg\" (UniqueName: \"kubernetes.io/projected/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-kube-api-access-2gvgg\") pod \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\" (UID: \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\") " Jul 16 17:50:24.357387 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:24.357365 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-extensions-socket-volume\") pod \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\" (UID: \"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07\") " Jul 16 17:50:24.357751 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:24.357726 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cbcb2caa-17e1-4e08-aa0c-e90036e6cd07" (UID: "cbcb2caa-17e1-4e08-aa0c-e90036e6cd07"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:24.359579 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:24.359556 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-kube-api-access-2gvgg" (OuterVolumeSpecName: "kube-api-access-2gvgg") pod "cbcb2caa-17e1-4e08-aa0c-e90036e6cd07" (UID: "cbcb2caa-17e1-4e08-aa0c-e90036e6cd07"). InnerVolumeSpecName "kube-api-access-2gvgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:50:24.457989 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:24.457952 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvgg\" (UniqueName: \"kubernetes.io/projected/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-kube-api-access-2gvgg\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:50:24.457989 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:24.457981 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07-extensions-socket-volume\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:50:25.047506 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.047472 2582 generic.go:356] "Generic (PLEG): container finished" podID="cbcb2caa-17e1-4e08-aa0c-e90036e6cd07" containerID="1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d" exitCode=0 Jul 16 17:50:25.047712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.047523 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" Jul 16 17:50:25.047712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.047548 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" event={"ID":"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07","Type":"ContainerDied","Data":"1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d"} Jul 16 17:50:25.047712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.047602 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8" event={"ID":"cbcb2caa-17e1-4e08-aa0c-e90036e6cd07","Type":"ContainerDied","Data":"31072dd8d7a814d8134ea4bed5ed38016882e7928b3d26f56565a220c977617c"} Jul 16 17:50:25.047712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.047624 2582 scope.go:117] "RemoveContainer" containerID="1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d" Jul 16 17:50:25.057425 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.057403 2582 scope.go:117] "RemoveContainer" containerID="1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d" Jul 16 17:50:25.057676 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:50:25.057657 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d\": container with ID starting with 1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d not found: ID does not exist" containerID="1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d" Jul 16 17:50:25.057740 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.057688 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d"} err="failed to get container status \"1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d\": rpc error: code = NotFound desc = could not find container \"1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d\": container with ID starting with 1673954a32906c77f8ce62af6627fd4db30c9145d92d37af80889a2ce69c476d not found: ID does not exist" Jul 16 17:50:25.072528 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.072496 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8"] Jul 16 17:50:25.078716 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:25.078693 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl7g8"] Jul 16 17:50:26.651133 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:50:26.651097 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbcb2caa-17e1-4e08-aa0c-e90036e6cd07" path="/var/lib/kubelet/pods/cbcb2caa-17e1-4e08-aa0c-e90036e6cd07/volumes" Jul 16 17:51:21.446101 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.446027 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28"] Jul 16 17:51:21.446495 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.446442 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcb2caa-17e1-4e08-aa0c-e90036e6cd07" containerName="manager" Jul 16 17:51:21.446495 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.446457 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcb2caa-17e1-4e08-aa0c-e90036e6cd07" containerName="manager" Jul 16 17:51:21.446566 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.446532 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcb2caa-17e1-4e08-aa0c-e90036e6cd07" containerName="manager" Jul 16 17:51:21.450713 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.450695 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:21.454142 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.454119 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8cgsj\"" Jul 16 17:51:21.464260 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.464233 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28"] Jul 16 17:51:21.567688 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.567647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzb6\" (UniqueName: \"kubernetes.io/projected/9d272945-fa69-48ca-88ef-43d0439890b4-kube-api-access-wmzb6\") pod \"kuadrant-operator-controller-manager-86878c769f-qzb28\" (UID: \"9d272945-fa69-48ca-88ef-43d0439890b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:21.567900 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.567713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d272945-fa69-48ca-88ef-43d0439890b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qzb28\" (UID: \"9d272945-fa69-48ca-88ef-43d0439890b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:21.669017 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.668978 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzb6\" (UniqueName: \"kubernetes.io/projected/9d272945-fa69-48ca-88ef-43d0439890b4-kube-api-access-wmzb6\") pod \"kuadrant-operator-controller-manager-86878c769f-qzb28\" (UID: \"9d272945-fa69-48ca-88ef-43d0439890b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:21.669195 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.669036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d272945-fa69-48ca-88ef-43d0439890b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qzb28\" (UID: \"9d272945-fa69-48ca-88ef-43d0439890b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:21.669447 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.669429 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d272945-fa69-48ca-88ef-43d0439890b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qzb28\" (UID: \"9d272945-fa69-48ca-88ef-43d0439890b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:21.680499 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.680463 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzb6\" (UniqueName: \"kubernetes.io/projected/9d272945-fa69-48ca-88ef-43d0439890b4-kube-api-access-wmzb6\") pod \"kuadrant-operator-controller-manager-86878c769f-qzb28\" (UID: \"9d272945-fa69-48ca-88ef-43d0439890b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:21.761345 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.761223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:21.921723 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:21.921698 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28"] Jul 16 17:51:21.922845 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:51:21.922825 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d272945_fa69_48ca_88ef_43d0439890b4.slice/crio-d7413ea01a12d55909d68a354537651d8bc08ac7d674594ff0bfb88de4932036 WatchSource:0}: Error finding container d7413ea01a12d55909d68a354537651d8bc08ac7d674594ff0bfb88de4932036: Status 404 returned error can't find the container with id d7413ea01a12d55909d68a354537651d8bc08ac7d674594ff0bfb88de4932036 Jul 16 17:51:22.367128 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:22.367084 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" event={"ID":"9d272945-fa69-48ca-88ef-43d0439890b4","Type":"ContainerStarted","Data":"d616d80a5a5f87cb224ba87a1f2c8e2ab77b6af7dbc3d7da1c4377f8a6b487e3"} Jul 16 17:51:22.367340 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:22.367142 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" event={"ID":"9d272945-fa69-48ca-88ef-43d0439890b4","Type":"ContainerStarted","Data":"d7413ea01a12d55909d68a354537651d8bc08ac7d674594ff0bfb88de4932036"} Jul 16 17:51:22.367340 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:22.367181 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:51:22.392562 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:22.392499 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" podStartSLOduration=1.392479948 podStartE2EDuration="1.392479948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:51:22.390846899 +0000 UTC m=+1344.142486457" watchObservedRunningTime="2026-07-16 17:51:22.392479948 +0000 UTC m=+1344.144119508" Jul 16 17:51:33.374787 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:51:33.374753 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qzb28" Jul 16 17:57:55.965059 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:55.965022 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85"] Jul 16 17:57:55.969049 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:55.969030 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:55.972292 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:55.972261 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8b0f3b27-openshift-default-dockercfg-shrdl\"" Jul 16 17:57:56.033505 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.033471 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85"] Jul 16 17:57:56.090293 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090247 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-certs\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.090470 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-token\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.090470 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-credential-socket\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.090470 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-podinfo\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.090470 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-data\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.090687 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-envoy\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.090687 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ed433a-e643-4c55-ae53-983cfbbfef9a-istiod-ca-cert\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.090687 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-socket\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.090817 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.090696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sksl\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-kube-api-access-5sksl\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191561 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-envoy\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191561 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191507 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ed433a-e643-4c55-ae53-983cfbbfef9a-istiod-ca-cert\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191561 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-socket\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191561 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sksl\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-kube-api-access-5sksl\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191900 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-certs\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191900 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-token\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191900 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-credential-socket\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191900 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-podinfo\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.191900 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.191711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-data\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.192143 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.192000 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-socket\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.192143 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.192075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-data\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.192143 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.192110 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-certs\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.192333 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.192171 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-credential-socket\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.192550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.192528 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ed433a-e643-4c55-ae53-983cfbbfef9a-istiod-ca-cert\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.194260 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.194237 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-envoy\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.194423 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.194404 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-podinfo\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.205843 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.205817 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-token\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.206450 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.206432 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sksl\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-kube-api-access-5sksl\") pod \"e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.279099 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.279060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:56.449025 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.448998 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85"] Jul 16 17:57:56.450589 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:57:56.450567 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ed433a_e643_4c55_ae53_983cfbbfef9a.slice/crio-10b3590850d34ea9e3473784265819c8d3d5c7ab5891f4cd19de67ab8afdd97e WatchSource:0}: Error finding container 10b3590850d34ea9e3473784265819c8d3d5c7ab5891f4cd19de67ab8afdd97e: Status 404 returned error can't find the container with id 10b3590850d34ea9e3473784265819c8d3d5c7ab5891f4cd19de67ab8afdd97e Jul 16 17:57:56.452636 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.452612 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:57:56.452705 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.452677 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:57:56.452741 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.452705 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:57:56.571696 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.571656 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" event={"ID":"32ed433a-e643-4c55-ae53-983cfbbfef9a","Type":"ContainerStarted","Data":"1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6"} Jul 16 17:57:56.571884 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.571721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" event={"ID":"32ed433a-e643-4c55-ae53-983cfbbfef9a","Type":"ContainerStarted","Data":"10b3590850d34ea9e3473784265819c8d3d5c7ab5891f4cd19de67ab8afdd97e"} Jul 16 17:57:56.623946 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:56.623896 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" podStartSLOduration=1.6238798509999999 podStartE2EDuration="1.623879851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:56.622434238 +0000 UTC m=+1738.374073796" watchObservedRunningTime="2026-07-16 17:57:56.623879851 +0000 UTC m=+1738.375519466" Jul 16 17:57:57.279939 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:57.279897 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:57.286437 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:57.286404 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:57.577635 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:57.577543 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:57:57.578729 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:57:57.578704 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:58:09.645067 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:09.645034 2582 generic.go:356] "Generic (PLEG): container finished" podID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerID="1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6" exitCode=0 Jul 16 17:58:09.645575 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:09.645103 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" event={"ID":"32ed433a-e643-4c55-ae53-983cfbbfef9a","Type":"ContainerDied","Data":"1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6"} Jul 16 17:58:09.645575 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:09.645476 2582 scope.go:117] "RemoveContainer" containerID="1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6" Jul 16 17:58:09.646483 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:09.646457 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:09.646564 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:09.646522 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:09.646564 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:09.646553 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:10.651914 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:10.651878 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" event={"ID":"32ed433a-e643-4c55-ae53-983cfbbfef9a","Type":"ContainerStarted","Data":"a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f"} Jul 16 17:58:11.279869 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:11.279827 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:58:11.280979 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:11.280950 2582 patch_prober.go:28] interesting pod/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 16 17:58:11.281210 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:11.281185 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 16 17:58:12.281817 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:12.281777 2582 patch_prober.go:28] interesting pod/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 16 17:58:12.282944 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:12.282507 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 16 17:58:12.577859 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:12.577769 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:58:13.280496 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:13.280457 2582 patch_prober.go:28] interesting pod/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 16 17:58:13.280803 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:13.280765 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 16 17:58:14.280486 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:14.280447 2582 patch_prober.go:28] interesting pod/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 16 17:58:14.281094 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:14.281053 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 16 17:58:15.284299 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:15.284249 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:58:15.285579 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:15.285555 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:58:17.815798 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:17.815756 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85"] Jul 16 17:58:17.816198 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:17.815957 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" containerID="cri-o://a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f" gracePeriod=30 Jul 16 17:58:23.066892 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.066866 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:58:23.147452 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147425 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-token\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147462 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-certs\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147524 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sksl\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-kube-api-access-5sksl\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147571 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ed433a-e643-4c55-ae53-983cfbbfef9a-istiod-ca-cert\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147634 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147629 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-credential-socket\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147864 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147676 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-data\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147864 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147715 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-podinfo\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147864 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147733 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-envoy\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147864 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147753 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-socket\") pod \"32ed433a-e643-4c55-ae53-983cfbbfef9a\" (UID: \"32ed433a-e643-4c55-ae53-983cfbbfef9a\") " Jul 16 17:58:23.147864 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.147844 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:58:23.148170 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.148075 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.148299 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.148255 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:58:23.148555 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.148531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ed433a-e643-4c55-ae53-983cfbbfef9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:58:23.148836 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.148805 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:58:23.148836 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.148819 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:58:23.150554 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.150509 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:58:23.150554 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.150533 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-kube-api-access-5sksl" (OuterVolumeSpecName: "kube-api-access-5sksl") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "kube-api-access-5sksl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:58:23.150711 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.150598 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:58:23.150850 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.150833 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32ed433a-e643-4c55-ae53-983cfbbfef9a" (UID: "32ed433a-e643-4c55-ae53-983cfbbfef9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:58:23.248641 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.248606 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sksl\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-kube-api-access-5sksl\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.248641 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.248635 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ed433a-e643-4c55-ae53-983cfbbfef9a-istiod-ca-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.248641 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.248644 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-credential-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.248870 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.248652 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-data\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.248870 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.248661 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-podinfo\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.248870 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.248669 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-envoy\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.248870 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.248678 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ed433a-e643-4c55-ae53-983cfbbfef9a-workload-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.248870 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.248685 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ed433a-e643-4c55-ae53-983cfbbfef9a-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:58:23.720206 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.720170 2582 generic.go:356] "Generic (PLEG): container finished" podID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerID="a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f" exitCode=0 Jul 16 17:58:23.720400 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.720218 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" Jul 16 17:58:23.720400 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.720265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" event={"ID":"32ed433a-e643-4c55-ae53-983cfbbfef9a","Type":"ContainerDied","Data":"a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f"} Jul 16 17:58:23.720400 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.720335 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85" event={"ID":"32ed433a-e643-4c55-ae53-983cfbbfef9a","Type":"ContainerDied","Data":"10b3590850d34ea9e3473784265819c8d3d5c7ab5891f4cd19de67ab8afdd97e"} Jul 16 17:58:23.720400 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.720358 2582 scope.go:117] "RemoveContainer" containerID="a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f" Jul 16 17:58:23.730562 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.730545 2582 scope.go:117] "RemoveContainer" containerID="1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6" Jul 16 17:58:23.740029 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.740009 2582 scope.go:117] "RemoveContainer" containerID="a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f" Jul 16 17:58:23.740330 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:58:23.740310 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f\": container with ID starting with a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f not found: ID does not exist" containerID="a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f" Jul 16 17:58:23.740385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.740339 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f"} err="failed to get container status \"a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f\": rpc error: code = NotFound desc = could not find container \"a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f\": container with ID starting with a9adfaf892763d3ce9b2ce207b8166fedced66ff075c2f4178919dfdd2b8317f not found: ID does not exist" Jul 16 17:58:23.740385 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.740358 2582 scope.go:117] "RemoveContainer" containerID="1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6" Jul 16 17:58:23.740611 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:58:23.740592 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6\": container with ID starting with 1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6 not found: ID does not exist" containerID="1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6" Jul 16 17:58:23.740665 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.740621 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6"} err="failed to get container status \"1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6\": rpc error: code = NotFound desc = could not find container \"1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6\": container with ID starting with 1ef8966a2073c18fef6c9e2cdbb7f27eb070bb8a1e36aa322766b1aa0646dca6 not found: ID does not exist" Jul 16 17:58:23.750124 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.750095 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85"] Jul 16 17:58:23.758141 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:23.758116 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8b0f3b27-openshift-default-78b488bfd7-tzx85"] Jul 16 17:58:24.651895 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:24.651864 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" path="/var/lib/kubelet/pods/32ed433a-e643-4c55-ae53-983cfbbfef9a/volumes" Jul 16 17:58:25.980773 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:25.980737 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8"] Jul 16 17:58:25.981350 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:25.981330 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" Jul 16 17:58:25.981438 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:25.981353 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" Jul 16 17:58:25.981490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:25.981479 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" Jul 16 17:58:25.981531 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:25.981491 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" Jul 16 17:58:25.986417 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:25.986393 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:25.992096 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:25.992071 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8c8af3ec-gw-openshift-default-dockercfg-s4ppc\"" Jul 16 17:58:26.001326 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.001301 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8"] Jul 16 17:58:26.074799 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.074754 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-socket\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.074987 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.074810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-envoy\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.074987 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.074867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdk2\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-kube-api-access-lvdk2\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.074987 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.074907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889b2572-7121-4eb9-8af4-a45242943d63-istio-podinfo\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.074987 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.074939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-certs\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.075204 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.075020 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889b2572-7121-4eb9-8af4-a45242943d63-istiod-ca-cert\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.075204 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.075065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-credential-socket\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.075204 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.075102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-istio-token\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.075204 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.075139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-data\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176072 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-envoy\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176263 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdk2\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-kube-api-access-lvdk2\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176263 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176105 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889b2572-7121-4eb9-8af4-a45242943d63-istio-podinfo\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176263 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-certs\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176263 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889b2572-7121-4eb9-8af4-a45242943d63-istiod-ca-cert\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176503 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-credential-socket\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176503 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-istio-token\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176503 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176442 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-data\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176503 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-socket\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176710 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-certs\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176768 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176730 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-data\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176812 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-socket\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.176885 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.176867 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-credential-socket\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.177234 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.177212 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889b2572-7121-4eb9-8af4-a45242943d63-istiod-ca-cert\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.178893 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.178866 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-envoy\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.179188 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.179168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889b2572-7121-4eb9-8af4-a45242943d63-istio-podinfo\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.189414 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.189390 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-istio-token\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.189536 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.189426 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdk2\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-kube-api-access-lvdk2\") pod \"e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.298387 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.298359 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:26.449910 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.449884 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8"] Jul 16 17:58:26.452251 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:58:26.452225 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889b2572_7121_4eb9_8af4_a45242943d63.slice/crio-6fb695d0dab12c344808682ab7bf78a630aa1504fe74378ea25ce336d6f185b9 WatchSource:0}: Error finding container 6fb695d0dab12c344808682ab7bf78a630aa1504fe74378ea25ce336d6f185b9: Status 404 returned error can't find the container with id 6fb695d0dab12c344808682ab7bf78a630aa1504fe74378ea25ce336d6f185b9 Jul 16 17:58:26.456837 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.456741 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:26.456837 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.456827 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:26.456924 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.456870 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:26.751490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.751397 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" event={"ID":"889b2572-7121-4eb9-8af4-a45242943d63","Type":"ContainerStarted","Data":"05af1c4e40fa35946c2fd445fbb71d14e9e140467a2fb70ec4756c50f420d497"} Jul 16 17:58:26.751490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.751464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" event={"ID":"889b2572-7121-4eb9-8af4-a45242943d63","Type":"ContainerStarted","Data":"6fb695d0dab12c344808682ab7bf78a630aa1504fe74378ea25ce336d6f185b9"} Jul 16 17:58:26.784474 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:26.784414 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" podStartSLOduration=1.784394687 podStartE2EDuration="1.784394687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:58:26.783250816 +0000 UTC m=+1768.534890374" watchObservedRunningTime="2026-07-16 17:58:26.784394687 +0000 UTC m=+1768.536034247" Jul 16 17:58:27.299369 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.299331 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:27.304354 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.304327 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:27.468114 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.468083 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv"] Jul 16 17:58:27.468495 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.468481 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" Jul 16 17:58:27.468548 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.468496 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ed433a-e643-4c55-ae53-983cfbbfef9a" containerName="istio-proxy" Jul 16 17:58:27.471785 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.471770 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.475579 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.475554 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nsr52\"" Jul 16 17:58:27.475702 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.475686 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8c8af3ec-serving-cert\"" Jul 16 17:58:27.486954 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.486922 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv"] Jul 16 17:58:27.589061 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.588963 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4tj\" (UniqueName: \"kubernetes.io/projected/462ebc3f-cbc8-46e7-a286-6153c4bd000f-kube-api-access-6v4tj\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.589061 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.589030 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.589061 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.589058 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ebc3f-cbc8-46e7-a286-6153c4bd000f-openshift-service-ca\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.689926 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.689887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4tj\" (UniqueName: \"kubernetes.io/projected/462ebc3f-cbc8-46e7-a286-6153c4bd000f-kube-api-access-6v4tj\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.690123 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.689968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.690123 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.690030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ebc3f-cbc8-46e7-a286-6153c4bd000f-openshift-service-ca\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.690123 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:58:27.690054 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-serving-cert: secret "maas-api-e2e-migrate-8c8af3ec-serving-cert" not found Jul 16 17:58:27.690123 ip-10-0-136-229 kubenswrapper[2582]: E0716 17:58:27.690112 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls podName:462ebc3f-cbc8-46e7-a286-6153c4bd000f nodeName:}" failed. No retries permitted until 2026-07-16 17:58:28.190092077 +0000 UTC m=+1769.941731637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls") pod "maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" (UID: "462ebc3f-cbc8-46e7-a286-6153c4bd000f") : secret "maas-api-e2e-migrate-8c8af3ec-serving-cert" not found Jul 16 17:58:27.690689 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.690671 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ebc3f-cbc8-46e7-a286-6153c4bd000f-openshift-service-ca\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.701909 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.701884 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4tj\" (UniqueName: \"kubernetes.io/projected/462ebc3f-cbc8-46e7-a286-6153c4bd000f-kube-api-access-6v4tj\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:27.757809 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.757774 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:27.758782 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:27.758764 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:28.195482 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:28.195378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:28.197993 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:28.197964 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls\") pod \"maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:28.385810 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:28.385773 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:28.545203 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:28.545177 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv"] Jul 16 17:58:28.546609 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:58:28.546577 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462ebc3f_cbc8_46e7_a286_6153c4bd000f.slice/crio-7dc608e457a649e7d291f1deda0100bb26b88cd2deff952185307f086ab8f271 WatchSource:0}: Error finding container 7dc608e457a649e7d291f1deda0100bb26b88cd2deff952185307f086ab8f271: Status 404 returned error can't find the container with id 7dc608e457a649e7d291f1deda0100bb26b88cd2deff952185307f086ab8f271 Jul 16 17:58:28.547943 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:28.547924 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:58:28.764670 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:28.764576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" event={"ID":"462ebc3f-cbc8-46e7-a286-6153c4bd000f","Type":"ContainerStarted","Data":"7dc608e457a649e7d291f1deda0100bb26b88cd2deff952185307f086ab8f271"} Jul 16 17:58:29.771744 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:29.771708 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" event={"ID":"462ebc3f-cbc8-46e7-a286-6153c4bd000f","Type":"ContainerStarted","Data":"bee27d53b09a787902ab0e810f9257307e5daad906f2f22632ab2d9457b57d2b"} Jul 16 17:58:29.772141 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:29.771806 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:29.795962 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:29.795900 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" podStartSLOduration=2.445597499 podStartE2EDuration="2.795879155s" totalImagesPullingTime="350.281656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:58:29.792049595 +0000 UTC m=+1771.543689155" watchObservedRunningTime="2026-07-16 17:58:29.795879155 +0000 UTC m=+1771.547518712" Jul 16 17:58:35.782000 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:35.781969 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:58:43.800490 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:43.800443 2582 patch_prober.go:28] interesting pod/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 17:58:43.802946 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:43.800526 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 17:58:44.857077 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:44.857046 2582 generic.go:356] "Generic (PLEG): container finished" podID="889b2572-7121-4eb9-8af4-a45242943d63" containerID="05af1c4e40fa35946c2fd445fbb71d14e9e140467a2fb70ec4756c50f420d497" exitCode=0 Jul 16 17:58:44.857488 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:44.857119 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" event={"ID":"889b2572-7121-4eb9-8af4-a45242943d63","Type":"ContainerDied","Data":"05af1c4e40fa35946c2fd445fbb71d14e9e140467a2fb70ec4756c50f420d497"} Jul 16 17:58:44.857532 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:44.857519 2582 scope.go:117] "RemoveContainer" containerID="05af1c4e40fa35946c2fd445fbb71d14e9e140467a2fb70ec4756c50f420d497" Jul 16 17:58:44.858473 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:44.858449 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:44.858542 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:44.858512 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:44.858577 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:44.858546 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:58:45.864307 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:45.864258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" event={"ID":"889b2572-7121-4eb9-8af4-a45242943d63","Type":"ContainerStarted","Data":"e5f5fc83ddc6b96022fb4aed95f9eff9659016af578d4d626844532e1a03951d"} Jul 16 17:58:46.299510 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:46.299467 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:46.299733 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:46.299652 2582 patch_prober.go:28] interesting pod/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 16 17:58:46.299733 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:46.299702 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 16 17:58:47.299712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:47.299214 2582 patch_prober.go:28] interesting pod/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 16 17:58:47.299712 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:47.299303 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 16 17:58:48.298993 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:48.298949 2582 patch_prober.go:28] interesting pod/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 16 17:58:48.299348 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:48.299315 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 16 17:58:49.299757 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:49.299712 2582 patch_prober.go:28] interesting pod/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 16 17:58:49.300265 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:49.299780 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 16 17:58:50.302125 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:50.302095 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:50.302550 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:50.302384 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:58:50.303037 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:58:50.303015 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:59:31.986973 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:31.986939 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv"] Jul 16 17:59:31.987472 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:31.987174 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" podUID="462ebc3f-cbc8-46e7-a286-6153c4bd000f" containerName="maas-api" containerID="cri-o://bee27d53b09a787902ab0e810f9257307e5daad906f2f22632ab2d9457b57d2b" gracePeriod=30 Jul 16 17:59:32.034479 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.034441 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8"] Jul 16 17:59:32.034743 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.034701 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" containerID="cri-o://e5f5fc83ddc6b96022fb4aed95f9eff9659016af578d4d626844532e1a03951d" gracePeriod=30 Jul 16 17:59:32.124635 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.124605 2582 generic.go:356] "Generic (PLEG): container finished" podID="462ebc3f-cbc8-46e7-a286-6153c4bd000f" containerID="bee27d53b09a787902ab0e810f9257307e5daad906f2f22632ab2d9457b57d2b" exitCode=0 Jul 16 17:59:32.124799 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.124656 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" event={"ID":"462ebc3f-cbc8-46e7-a286-6153c4bd000f","Type":"ContainerDied","Data":"bee27d53b09a787902ab0e810f9257307e5daad906f2f22632ab2d9457b57d2b"} Jul 16 17:59:32.245976 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.245906 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:59:32.421601 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.421565 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v4tj\" (UniqueName: \"kubernetes.io/projected/462ebc3f-cbc8-46e7-a286-6153c4bd000f-kube-api-access-6v4tj\") pod \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " Jul 16 17:59:32.421601 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.421600 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls\") pod \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " Jul 16 17:59:32.421859 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.421631 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ebc3f-cbc8-46e7-a286-6153c4bd000f-openshift-service-ca\") pod \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\" (UID: \"462ebc3f-cbc8-46e7-a286-6153c4bd000f\") " Jul 16 17:59:32.422099 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.422074 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462ebc3f-cbc8-46e7-a286-6153c4bd000f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "462ebc3f-cbc8-46e7-a286-6153c4bd000f" (UID: "462ebc3f-cbc8-46e7-a286-6153c4bd000f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:59:32.424154 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.424124 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "462ebc3f-cbc8-46e7-a286-6153c4bd000f" (UID: "462ebc3f-cbc8-46e7-a286-6153c4bd000f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:59:32.424295 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.424250 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462ebc3f-cbc8-46e7-a286-6153c4bd000f-kube-api-access-6v4tj" (OuterVolumeSpecName: "kube-api-access-6v4tj") pod "462ebc3f-cbc8-46e7-a286-6153c4bd000f" (UID: "462ebc3f-cbc8-46e7-a286-6153c4bd000f"). InnerVolumeSpecName "kube-api-access-6v4tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:32.522883 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.522781 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4tj\" (UniqueName: \"kubernetes.io/projected/462ebc3f-cbc8-46e7-a286-6153c4bd000f-kube-api-access-6v4tj\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:32.522883 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.522824 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ebc3f-cbc8-46e7-a286-6153c4bd000f-maas-api-tls\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:32.522883 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:32.522834 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ebc3f-cbc8-46e7-a286-6153c4bd000f-openshift-service-ca\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:33.131518 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:33.131486 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" Jul 16 17:59:33.131518 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:33.131504 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv" event={"ID":"462ebc3f-cbc8-46e7-a286-6153c4bd000f","Type":"ContainerDied","Data":"7dc608e457a649e7d291f1deda0100bb26b88cd2deff952185307f086ab8f271"} Jul 16 17:59:33.132013 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:33.131558 2582 scope.go:117] "RemoveContainer" containerID="bee27d53b09a787902ab0e810f9257307e5daad906f2f22632ab2d9457b57d2b" Jul 16 17:59:33.159254 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:33.159217 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv"] Jul 16 17:59:33.169614 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:33.169584 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8c8af3ec-644f9d75cb-5ktrv"] Jul 16 17:59:34.652171 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:34.652134 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462ebc3f-cbc8-46e7-a286-6153c4bd000f" path="/var/lib/kubelet/pods/462ebc3f-cbc8-46e7-a286-6153c4bd000f/volumes" Jul 16 17:59:35.305224 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:35.305188 2582 patch_prober.go:28] interesting pod/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 17:59:35.305412 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:35.305258 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:59:37.156769 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.156733 2582 generic.go:356] "Generic (PLEG): container finished" podID="889b2572-7121-4eb9-8af4-a45242943d63" containerID="e5f5fc83ddc6b96022fb4aed95f9eff9659016af578d4d626844532e1a03951d" exitCode=0 Jul 16 17:59:37.157129 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.156797 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" event={"ID":"889b2572-7121-4eb9-8af4-a45242943d63","Type":"ContainerDied","Data":"e5f5fc83ddc6b96022fb4aed95f9eff9659016af578d4d626844532e1a03951d"} Jul 16 17:59:37.157129 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.156868 2582 scope.go:117] "RemoveContainer" containerID="05af1c4e40fa35946c2fd445fbb71d14e9e140467a2fb70ec4756c50f420d497" Jul 16 17:59:37.302713 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.302657 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:59:37.465930 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.465899 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889b2572-7121-4eb9-8af4-a45242943d63-istio-podinfo\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.465930 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.465937 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-certs\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.466176 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.465975 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-credential-socket\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.466176 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466015 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889b2572-7121-4eb9-8af4-a45242943d63-istiod-ca-cert\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.466176 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466046 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-envoy\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.466176 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466070 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvdk2\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-kube-api-access-lvdk2\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.466176 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466127 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-socket\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.466176 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466154 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-istio-token\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.466674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466186 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-data\") pod \"889b2572-7121-4eb9-8af4-a45242943d63\" (UID: \"889b2572-7121-4eb9-8af4-a45242943d63\") " Jul 16 17:59:37.466674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466356 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:37.466674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466373 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:37.466674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466421 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889b2572-7121-4eb9-8af4-a45242943d63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:59:37.466674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466559 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889b2572-7121-4eb9-8af4-a45242943d63-istiod-ca-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:37.466674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466576 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:37.466674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466590 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-credential-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:37.466674 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.466628 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:37.467260 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.467060 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-data" (OuterVolumeSpecName: "istio-data") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:37.468585 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.468552 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-kube-api-access-lvdk2" (OuterVolumeSpecName: "kube-api-access-lvdk2") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "kube-api-access-lvdk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:37.468585 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.468569 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:37.468803 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.468787 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-istio-token" (OuterVolumeSpecName: "istio-token") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:37.468880 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.468864 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/889b2572-7121-4eb9-8af4-a45242943d63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "889b2572-7121-4eb9-8af4-a45242943d63" (UID: "889b2572-7121-4eb9-8af4-a45242943d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:59:37.567847 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.567751 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-envoy\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:37.567847 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.567797 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdk2\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-kube-api-access-lvdk2\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:37.567847 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.567808 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-workload-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:37.567847 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.567818 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889b2572-7121-4eb9-8af4-a45242943d63-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:37.567847 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.567827 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889b2572-7121-4eb9-8af4-a45242943d63-istio-data\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:37.567847 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:37.567834 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889b2572-7121-4eb9-8af4-a45242943d63-istio-podinfo\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 17:59:38.163637 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:38.163606 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" Jul 16 17:59:38.163637 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:38.163625 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8" event={"ID":"889b2572-7121-4eb9-8af4-a45242943d63","Type":"ContainerDied","Data":"6fb695d0dab12c344808682ab7bf78a630aa1504fe74378ea25ce336d6f185b9"} Jul 16 17:59:38.164195 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:38.163681 2582 scope.go:117] "RemoveContainer" containerID="e5f5fc83ddc6b96022fb4aed95f9eff9659016af578d4d626844532e1a03951d" Jul 16 17:59:38.197573 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:38.197526 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8"] Jul 16 17:59:38.203156 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:38.203130 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8c8af3ec-gw-openshift-default-6d6bd479c9-ntvw8"] Jul 16 17:59:38.651846 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:38.651804 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="889b2572-7121-4eb9-8af4-a45242943d63" path="/var/lib/kubelet/pods/889b2572-7121-4eb9-8af4-a45242943d63/volumes" Jul 16 17:59:40.044748 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.044718 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc"] Jul 16 17:59:40.045117 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045105 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462ebc3f-cbc8-46e7-a286-6153c4bd000f" containerName="maas-api" Jul 16 17:59:40.045166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045119 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="462ebc3f-cbc8-46e7-a286-6153c4bd000f" containerName="maas-api" Jul 16 17:59:40.045166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045141 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" Jul 16 17:59:40.045166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045146 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" Jul 16 17:59:40.045166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045152 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" Jul 16 17:59:40.045166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045158 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" Jul 16 17:59:40.045356 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045213 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="462ebc3f-cbc8-46e7-a286-6153c4bd000f" containerName="maas-api" Jul 16 17:59:40.045356 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045222 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" Jul 16 17:59:40.045356 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.045228 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="889b2572-7121-4eb9-8af4-a45242943d63" containerName="istio-proxy" Jul 16 17:59:40.049697 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.049680 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.053011 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.052985 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-41fe79b5-openshift-default-dockercfg-nkgq2\"" Jul 16 17:59:40.064027 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.063996 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc"] Jul 16 17:59:40.193919 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.193887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-socket\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.193919 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.193921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-certs\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.193919 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.193939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8986fb92-add0-473d-b58f-1de22634844f-istiod-ca-cert\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.194184 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.194001 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-credential-socket\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.194184 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.194030 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8986fb92-add0-473d-b58f-1de22634844f-istio-podinfo\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.194184 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.194049 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvl8n\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-kube-api-access-zvl8n\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.194184 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.194065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-istio-token\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.194184 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.194083 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-envoy\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.194184 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.194104 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-data\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.295549 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-socket\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.295549 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-certs\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.295778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8986fb92-add0-473d-b58f-1de22634844f-istiod-ca-cert\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.295778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295619 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-credential-socket\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.295778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8986fb92-add0-473d-b58f-1de22634844f-istio-podinfo\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.295778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295696 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvl8n\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-kube-api-access-zvl8n\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.295778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295722 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-istio-token\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.295778 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-envoy\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.296172 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.295792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-data\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.296172 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.296008 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-socket\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.296172 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.296076 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-certs\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.296172 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.296104 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-data\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.296172 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.296099 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-credential-socket\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.296710 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.296687 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8986fb92-add0-473d-b58f-1de22634844f-istiod-ca-cert\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.298342 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.298320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-envoy\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.298462 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.298443 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8986fb92-add0-473d-b58f-1de22634844f-istio-podinfo\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.308691 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.308579 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvl8n\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-kube-api-access-zvl8n\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.310262 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.310238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-istio-token\") pod \"e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.361034 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.360990 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:40.521251 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.521224 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc"] Jul 16 17:59:40.522622 ip-10-0-136-229 kubenswrapper[2582]: W0716 17:59:40.522589 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8986fb92_add0_473d_b58f_1de22634844f.slice/crio-77e5fc3cabf08c537fc5651ec5a47352b542f7c992d23da9485a41bc05d27f18 WatchSource:0}: Error finding container 77e5fc3cabf08c537fc5651ec5a47352b542f7c992d23da9485a41bc05d27f18: Status 404 returned error can't find the container with id 77e5fc3cabf08c537fc5651ec5a47352b542f7c992d23da9485a41bc05d27f18 Jul 16 17:59:40.524637 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.524606 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:59:40.524731 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.524674 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:59:40.524731 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:40.524702 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:59:41.184166 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:41.184123 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" event={"ID":"8986fb92-add0-473d-b58f-1de22634844f","Type":"ContainerStarted","Data":"af5a36f14eb58d57605057c8a282b54d16f4f326e1c18e3e036f137c53c499e6"} Jul 16 17:59:41.184676 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:41.184193 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" event={"ID":"8986fb92-add0-473d-b58f-1de22634844f","Type":"ContainerStarted","Data":"77e5fc3cabf08c537fc5651ec5a47352b542f7c992d23da9485a41bc05d27f18"} Jul 16 17:59:41.212205 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:41.212154 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" podStartSLOduration=1.212137157 podStartE2EDuration="1.212137157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:59:41.209993458 +0000 UTC m=+1842.961633010" watchObservedRunningTime="2026-07-16 17:59:41.212137157 +0000 UTC m=+1842.963776793" Jul 16 17:59:41.361900 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:41.361861 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:41.367103 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:41.367077 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:42.190133 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:42.190101 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:42.191017 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:42.190998 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:56.269590 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:56.269555 2582 generic.go:356] "Generic (PLEG): container finished" podID="8986fb92-add0-473d-b58f-1de22634844f" containerID="af5a36f14eb58d57605057c8a282b54d16f4f326e1c18e3e036f137c53c499e6" exitCode=0 Jul 16 17:59:56.270014 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:56.269631 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" event={"ID":"8986fb92-add0-473d-b58f-1de22634844f","Type":"ContainerDied","Data":"af5a36f14eb58d57605057c8a282b54d16f4f326e1c18e3e036f137c53c499e6"} Jul 16 17:59:56.270077 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:56.270062 2582 scope.go:117] "RemoveContainer" containerID="af5a36f14eb58d57605057c8a282b54d16f4f326e1c18e3e036f137c53c499e6" Jul 16 17:59:56.270965 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:56.270942 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:59:56.271027 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:56.271014 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:59:56.271063 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:56.271049 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 17:59:57.190492 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:57.190392 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:57.287106 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:57.287065 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" event={"ID":"8986fb92-add0-473d-b58f-1de22634844f","Type":"ContainerStarted","Data":"8a9f6397cd27317225722fc00c692ba00b2c37a724cb53942b2f29cb37093bab"} Jul 16 17:59:57.361614 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:57.361584 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 17:59:57.362803 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:57.362776 2582 patch_prober.go:28] interesting pod/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 16 17:59:57.362943 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:57.362833 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 16 17:59:58.362086 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:58.362040 2582 patch_prober.go:28] interesting pod/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 16 17:59:58.362619 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:58.362104 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 16 17:59:59.362089 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:59.362055 2582 patch_prober.go:28] interesting pod/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 16 17:59:59.362507 ip-10-0-136-229 kubenswrapper[2582]: I0716 17:59:59.362107 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 16 18:00:00.361505 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:00.361460 2582 patch_prober.go:28] interesting pod/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 16 18:00:00.361687 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:00.361540 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 16 18:00:01.365879 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:01.365840 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 18:00:01.366235 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:01.366129 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 18:00:01.366796 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:01.366776 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 18:00:53.375001 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:53.374961 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc"] Jul 16 18:00:53.375520 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:53.375222 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" containerID="cri-o://8a9f6397cd27317225722fc00c692ba00b2c37a724cb53942b2f29cb37093bab" gracePeriod=30 Jul 16 18:00:58.627206 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.627174 2582 generic.go:356] "Generic (PLEG): container finished" podID="8986fb92-add0-473d-b58f-1de22634844f" containerID="8a9f6397cd27317225722fc00c692ba00b2c37a724cb53942b2f29cb37093bab" exitCode=0 Jul 16 18:00:58.627578 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.627212 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" event={"ID":"8986fb92-add0-473d-b58f-1de22634844f","Type":"ContainerDied","Data":"8a9f6397cd27317225722fc00c692ba00b2c37a724cb53942b2f29cb37093bab"} Jul 16 18:00:58.627578 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.627274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" event={"ID":"8986fb92-add0-473d-b58f-1de22634844f","Type":"ContainerDied","Data":"77e5fc3cabf08c537fc5651ec5a47352b542f7c992d23da9485a41bc05d27f18"} Jul 16 18:00:58.627578 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.627307 2582 scope.go:117] "RemoveContainer" containerID="af5a36f14eb58d57605057c8a282b54d16f4f326e1c18e3e036f137c53c499e6" Jul 16 18:00:58.627578 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.627325 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77e5fc3cabf08c537fc5651ec5a47352b542f7c992d23da9485a41bc05d27f18" Jul 16 18:00:58.645259 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:00:58.645232 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5a36f14eb58d57605057c8a282b54d16f4f326e1c18e3e036f137c53c499e6\": container with ID starting with af5a36f14eb58d57605057c8a282b54d16f4f326e1c18e3e036f137c53c499e6 not found: ID does not exist" containerID="af5a36f14eb58d57605057c8a282b54d16f4f326e1c18e3e036f137c53c499e6" Jul 16 18:00:58.648753 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.648725 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 18:00:58.674921 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.674888 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-istio-token\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675123 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.674947 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-certs\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675123 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.674971 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-socket\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675123 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.674995 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-envoy\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675123 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.675030 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8986fb92-add0-473d-b58f-1de22634844f-istiod-ca-cert\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675123 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.675067 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-credential-socket\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675428 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.675157 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8986fb92-add0-473d-b58f-1de22634844f-istio-podinfo\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675428 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.675202 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvl8n\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-kube-api-access-zvl8n\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675428 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.675235 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-data\") pod \"8986fb92-add0-473d-b58f-1de22634844f\" (UID: \"8986fb92-add0-473d-b58f-1de22634844f\") " Jul 16 18:00:58.675428 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.675252 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:00:58.675631 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.675556 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:58.676211 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.676168 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:00:58.676364 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.676238 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:00:58.676364 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.676127 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8986fb92-add0-473d-b58f-1de22634844f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:00:58.676530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.676505 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:00:58.678050 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.678011 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:00:58.678503 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.678478 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8986fb92-add0-473d-b58f-1de22634844f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 18:00:58.678841 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.678817 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-kube-api-access-zvl8n" (OuterVolumeSpecName: "kube-api-access-zvl8n") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "kube-api-access-zvl8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:00:58.678930 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.678843 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8986fb92-add0-473d-b58f-1de22634844f" (UID: "8986fb92-add0-473d-b58f-1de22634844f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:00:58.776891 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.776852 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-workload-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:58.776891 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.776878 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-envoy\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:58.776891 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.776891 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8986fb92-add0-473d-b58f-1de22634844f-istiod-ca-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:58.777503 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.776901 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-credential-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:58.777503 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.776915 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8986fb92-add0-473d-b58f-1de22634844f-istio-podinfo\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:58.777503 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.776928 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvl8n\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-kube-api-access-zvl8n\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:58.777503 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.776938 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8986fb92-add0-473d-b58f-1de22634844f-istio-data\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:58.777503 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:58.776949 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8986fb92-add0-473d-b58f-1de22634844f-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:00:59.633519 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:59.633490 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc" Jul 16 18:00:59.661579 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:59.661543 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc"] Jul 16 18:00:59.666144 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:00:59.666115 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-41fe79b5-openshift-default-8669fcb589-rfmqc"] Jul 16 18:01:00.549009 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.548977 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p"] Jul 16 18:01:00.549521 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.549504 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" Jul 16 18:01:00.549572 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.549524 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" Jul 16 18:01:00.549572 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.549538 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" Jul 16 18:01:00.549572 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.549545 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" Jul 16 18:01:00.549670 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.549631 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" Jul 16 18:01:00.552631 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.552615 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.555539 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.555515 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5410de53-openshift-default-dockercfg-qlsnk\"" Jul 16 18:01:00.568531 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.568501 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p"] Jul 16 18:01:00.593426 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-socket\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.593426 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4p5\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-kube-api-access-km4p5\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.593707 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-data\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.593707 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-certs\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.593707 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-envoy\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.593707 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-credential-socket\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.593707 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-istio-token\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.593707 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bed580-4baf-4a62-8906-7b893df62ba5-istiod-ca-cert\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.593925 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.593752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bed580-4baf-4a62-8906-7b893df62ba5-istio-podinfo\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.651395 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.651353 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8986fb92-add0-473d-b58f-1de22634844f" path="/var/lib/kubelet/pods/8986fb92-add0-473d-b58f-1de22634844f/volumes" Jul 16 18:01:00.694850 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.694812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-socket\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.694850 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.694852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4p5\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-kube-api-access-km4p5\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695107 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.694886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-data\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695107 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.694955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-certs\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695107 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.694985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-envoy\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695107 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-credential-socket\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695107 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-istio-token\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695107 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bed580-4baf-4a62-8906-7b893df62ba5-istiod-ca-cert\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695472 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bed580-4baf-4a62-8906-7b893df62ba5-istio-podinfo\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695472 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695383 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-socket\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695577 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695513 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-credential-socket\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695577 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695525 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-data\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.695577 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695536 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-certs\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.696000 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.695969 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bed580-4baf-4a62-8906-7b893df62ba5-istiod-ca-cert\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.697725 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.697701 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-envoy\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.697862 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.697845 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bed580-4baf-4a62-8906-7b893df62ba5-istio-podinfo\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.706870 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.706844 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-istio-token\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.706984 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.706919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4p5\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-kube-api-access-km4p5\") pod \"e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:00.863962 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:00.863871 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:01.024526 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.024500 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p"] Jul 16 18:01:01.026512 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:01:01.026480 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bed580_4baf_4a62_8906_7b893df62ba5.slice/crio-f99deb8c2da5cab048b347c696f58337a985984b564c49b45d17b0136b61b4ff WatchSource:0}: Error finding container f99deb8c2da5cab048b347c696f58337a985984b564c49b45d17b0136b61b4ff: Status 404 returned error can't find the container with id f99deb8c2da5cab048b347c696f58337a985984b564c49b45d17b0136b61b4ff Jul 16 18:01:01.028672 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.028644 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:01:01.028741 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.028707 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:01:01.028741 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.028734 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:01:01.646144 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.646109 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" event={"ID":"88bed580-4baf-4a62-8906-7b893df62ba5","Type":"ContainerStarted","Data":"1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa"} Jul 16 18:01:01.646327 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.646153 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" event={"ID":"88bed580-4baf-4a62-8906-7b893df62ba5","Type":"ContainerStarted","Data":"f99deb8c2da5cab048b347c696f58337a985984b564c49b45d17b0136b61b4ff"} Jul 16 18:01:01.671329 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.671256 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" podStartSLOduration=1.671238445 podStartE2EDuration="1.671238445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:01:01.669117169 +0000 UTC m=+1923.420756728" watchObservedRunningTime="2026-07-16 18:01:01.671238445 +0000 UTC m=+1923.422878003" Jul 16 18:01:01.864374 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.864338 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:01.908436 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.908355 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb"] Jul 16 18:01:01.908814 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.908801 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8986fb92-add0-473d-b58f-1de22634844f" containerName="istio-proxy" Jul 16 18:01:01.912011 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.911988 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:01.915229 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.915197 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nsr52\"" Jul 16 18:01:01.915532 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.915514 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5410de53-serving-cert\"" Jul 16 18:01:01.930087 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:01.930048 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb"] Jul 16 18:01:02.007744 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.007700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.007744 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.007747 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-openshift-service-ca\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.007987 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.007836 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcw48\" (UniqueName: \"kubernetes.io/projected/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-kube-api-access-dcw48\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.109239 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.109200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.109239 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.109238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-openshift-service-ca\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.109550 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.109270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcw48\" (UniqueName: \"kubernetes.io/projected/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-kube-api-access-dcw48\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.109550 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:01:02.109411 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-serving-cert: secret "maas-api-e2e-derive-5410de53-serving-cert" not found Jul 16 18:01:02.109550 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:01:02.109490 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls podName:bd9343db-ce76-4fb4-a6cc-aa81a458d83f nodeName:}" failed. No retries permitted until 2026-07-16 18:01:02.609467138 +0000 UTC m=+1924.361106677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls") pod "maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" (UID: "bd9343db-ce76-4fb4-a6cc-aa81a458d83f") : secret "maas-api-e2e-derive-5410de53-serving-cert" not found Jul 16 18:01:02.109906 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.109877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-openshift-service-ca\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.122874 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.122842 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcw48\" (UniqueName: \"kubernetes.io/projected/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-kube-api-access-dcw48\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.614531 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.614485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.617082 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.617061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls\") pod \"maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.833261 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.833218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:02.870025 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.869950 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:02.983907 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:02.983876 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb"] Jul 16 18:01:02.985808 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:01:02.985782 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9343db_ce76_4fb4_a6cc_aa81a458d83f.slice/crio-fed1351e64c394dc5aa37b0644ca3d4374c52c19389073d42fe36c2d68c98350 WatchSource:0}: Error finding container fed1351e64c394dc5aa37b0644ca3d4374c52c19389073d42fe36c2d68c98350: Status 404 returned error can't find the container with id fed1351e64c394dc5aa37b0644ca3d4374c52c19389073d42fe36c2d68c98350 Jul 16 18:01:03.658745 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:03.658710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" event={"ID":"bd9343db-ce76-4fb4-a6cc-aa81a458d83f","Type":"ContainerStarted","Data":"fed1351e64c394dc5aa37b0644ca3d4374c52c19389073d42fe36c2d68c98350"} Jul 16 18:01:03.658932 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:03.658915 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:03.659871 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:03.659856 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:04.666109 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:04.666071 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" event={"ID":"bd9343db-ce76-4fb4-a6cc-aa81a458d83f","Type":"ContainerStarted","Data":"7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f"} Jul 16 18:01:04.687759 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:04.687691 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" podStartSLOduration=2.817133458 podStartE2EDuration="3.687669146s" totalImagesPullingTime="870.535688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:01:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:01:04.686068672 +0000 UTC m=+1926.437708231" watchObservedRunningTime="2026-07-16 18:01:04.687669146 +0000 UTC m=+1926.439308705" Jul 16 18:01:05.671889 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:05.671852 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:11.683141 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:11.683101 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:01:13.734094 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:13.734059 2582 generic.go:356] "Generic (PLEG): container finished" podID="88bed580-4baf-4a62-8906-7b893df62ba5" containerID="1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa" exitCode=0 Jul 16 18:01:13.734539 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:13.734142 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" event={"ID":"88bed580-4baf-4a62-8906-7b893df62ba5","Type":"ContainerDied","Data":"1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa"} Jul 16 18:01:13.734620 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:13.734604 2582 scope.go:117] "RemoveContainer" containerID="1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa" Jul 16 18:01:13.738699 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:13.738657 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:01:13.738831 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:13.738753 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:01:13.738831 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:13.738793 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:01:14.741376 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:14.741345 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" event={"ID":"88bed580-4baf-4a62-8906-7b893df62ba5","Type":"ContainerStarted","Data":"e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2"} Jul 16 18:01:14.863977 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:14.863936 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:14.864987 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:14.864952 2582 patch_prober.go:28] interesting pod/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 16 18:01:14.865134 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:14.864997 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 16 18:01:15.864727 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:15.864692 2582 patch_prober.go:28] interesting pod/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 16 18:01:15.865116 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:15.864756 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 16 18:01:16.865204 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:16.865171 2582 patch_prober.go:28] interesting pod/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 16 18:01:16.865620 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:16.865236 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 16 18:01:17.864799 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:17.864763 2582 patch_prober.go:28] interesting pod/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 16 18:01:17.864981 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:17.864824 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 16 18:01:18.659038 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:18.659007 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:18.868001 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:18.867970 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:01:18.869038 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:01:18.869019 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:02:16.218459 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.218422 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb"] Jul 16 18:02:16.219517 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.219275 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" podUID="bd9343db-ce76-4fb4-a6cc-aa81a458d83f" containerName="maas-api" containerID="cri-o://7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f" gracePeriod=30 Jul 16 18:02:16.487131 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.487103 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:02:16.529343 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.529315 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls\") pod \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " Jul 16 18:02:16.529557 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.529389 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-openshift-service-ca\") pod \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " Jul 16 18:02:16.529557 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.529447 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcw48\" (UniqueName: \"kubernetes.io/projected/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-kube-api-access-dcw48\") pod \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\" (UID: \"bd9343db-ce76-4fb4-a6cc-aa81a458d83f\") " Jul 16 18:02:16.529797 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.529772 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd9343db-ce76-4fb4-a6cc-aa81a458d83f" (UID: "bd9343db-ce76-4fb4-a6cc-aa81a458d83f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:02:16.531721 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.531692 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-kube-api-access-dcw48" (OuterVolumeSpecName: "kube-api-access-dcw48") pod "bd9343db-ce76-4fb4-a6cc-aa81a458d83f" (UID: "bd9343db-ce76-4fb4-a6cc-aa81a458d83f"). InnerVolumeSpecName "kube-api-access-dcw48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:02:16.531721 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.531718 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd9343db-ce76-4fb4-a6cc-aa81a458d83f" (UID: "bd9343db-ce76-4fb4-a6cc-aa81a458d83f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:02:16.630554 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.630507 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-maas-api-tls\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:16.630554 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.630548 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-openshift-service-ca\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:16.630554 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:16.630559 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcw48\" (UniqueName: \"kubernetes.io/projected/bd9343db-ce76-4fb4-a6cc-aa81a458d83f-kube-api-access-dcw48\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:17.089496 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.089457 2582 generic.go:356] "Generic (PLEG): container finished" podID="bd9343db-ce76-4fb4-a6cc-aa81a458d83f" containerID="7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f" exitCode=0 Jul 16 18:02:17.089708 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.089515 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" Jul 16 18:02:17.089708 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.089514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" event={"ID":"bd9343db-ce76-4fb4-a6cc-aa81a458d83f","Type":"ContainerDied","Data":"7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f"} Jul 16 18:02:17.089708 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.089596 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb" event={"ID":"bd9343db-ce76-4fb4-a6cc-aa81a458d83f","Type":"ContainerDied","Data":"fed1351e64c394dc5aa37b0644ca3d4374c52c19389073d42fe36c2d68c98350"} Jul 16 18:02:17.089708 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.089617 2582 scope.go:117] "RemoveContainer" containerID="7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f" Jul 16 18:02:17.099352 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.099331 2582 scope.go:117] "RemoveContainer" containerID="7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f" Jul 16 18:02:17.099633 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:02:17.099610 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f\": container with ID starting with 7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f not found: ID does not exist" containerID="7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f" Jul 16 18:02:17.099722 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.099641 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f"} err="failed to get container status \"7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f\": rpc error: code = NotFound desc = could not find container \"7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f\": container with ID starting with 7bfdb9a19cc896754b4c72cd81d87b6eb379b3e0a7e19f3e3ad5896239c2398f not found: ID does not exist" Jul 16 18:02:17.112577 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.112551 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb"] Jul 16 18:02:17.117803 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:17.117779 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5410de53-6df6fb4fd8-bgrlb"] Jul 16 18:02:18.651551 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:18.651516 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd9343db-ce76-4fb4-a6cc-aa81a458d83f" path="/var/lib/kubelet/pods/bd9343db-ce76-4fb4-a6cc-aa81a458d83f/volumes" Jul 16 18:02:21.531871 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:21.531841 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p"] Jul 16 18:02:21.532268 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:21.532060 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" containerID="cri-o://e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2" gracePeriod=30 Jul 16 18:02:26.799614 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.799538 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:02:26.940077 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940015 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-istio-token\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940077 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940086 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-credential-socket\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940379 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940111 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-data\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940379 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940138 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bed580-4baf-4a62-8906-7b893df62ba5-istio-podinfo\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940379 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940159 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-socket\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940379 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940195 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-envoy\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940379 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940210 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bed580-4baf-4a62-8906-7b893df62ba5-istiod-ca-cert\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940379 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940250 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-certs\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940379 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940315 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km4p5\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-kube-api-access-km4p5\") pod \"88bed580-4baf-4a62-8906-7b893df62ba5\" (UID: \"88bed580-4baf-4a62-8906-7b893df62ba5\") " Jul 16 18:02:26.940739 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940481 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:02:26.940739 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940604 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-credential-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:26.940739 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940673 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:02:26.940739 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940725 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:02:26.940962 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.940759 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bed580-4baf-4a62-8906-7b893df62ba5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:02:26.941450 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.941419 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-data" (OuterVolumeSpecName: "istio-data") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:02:26.942579 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.942547 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-istio-token" (OuterVolumeSpecName: "istio-token") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:02:26.942722 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.942703 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-kube-api-access-km4p5" (OuterVolumeSpecName: "kube-api-access-km4p5") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "kube-api-access-km4p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:02:26.942797 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.942754 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:02:26.942843 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:26.942831 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88bed580-4baf-4a62-8906-7b893df62ba5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88bed580-4baf-4a62-8906-7b893df62ba5" (UID: "88bed580-4baf-4a62-8906-7b893df62ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 18:02:27.041248 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.041203 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-data\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:27.041248 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.041232 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bed580-4baf-4a62-8906-7b893df62ba5-istio-podinfo\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:27.041248 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.041241 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:27.041248 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.041250 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-istio-envoy\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:27.041248 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.041258 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bed580-4baf-4a62-8906-7b893df62ba5-istiod-ca-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:27.041567 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.041266 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bed580-4baf-4a62-8906-7b893df62ba5-workload-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:27.041567 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.041273 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4p5\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-kube-api-access-km4p5\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:27.041567 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.041312 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bed580-4baf-4a62-8906-7b893df62ba5-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:02:27.149627 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.149532 2582 generic.go:356] "Generic (PLEG): container finished" podID="88bed580-4baf-4a62-8906-7b893df62ba5" containerID="e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2" exitCode=0 Jul 16 18:02:27.149627 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.149581 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" Jul 16 18:02:27.149860 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.149618 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" event={"ID":"88bed580-4baf-4a62-8906-7b893df62ba5","Type":"ContainerDied","Data":"e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2"} Jul 16 18:02:27.149860 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.149683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p" event={"ID":"88bed580-4baf-4a62-8906-7b893df62ba5","Type":"ContainerDied","Data":"f99deb8c2da5cab048b347c696f58337a985984b564c49b45d17b0136b61b4ff"} Jul 16 18:02:27.149860 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.149716 2582 scope.go:117] "RemoveContainer" containerID="e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2" Jul 16 18:02:27.159450 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.159432 2582 scope.go:117] "RemoveContainer" containerID="1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa" Jul 16 18:02:27.167982 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.167967 2582 scope.go:117] "RemoveContainer" containerID="e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2" Jul 16 18:02:27.168265 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:02:27.168248 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2\": container with ID starting with e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2 not found: ID does not exist" containerID="e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2" Jul 16 18:02:27.168350 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.168274 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2"} err="failed to get container status \"e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2\": rpc error: code = NotFound desc = could not find container \"e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2\": container with ID starting with e25492dcae3c712c2729f892ede9b48f0c79bb6627fb0ed0cef5a4c6d655b3d2 not found: ID does not exist" Jul 16 18:02:27.168350 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.168316 2582 scope.go:117] "RemoveContainer" containerID="1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa" Jul 16 18:02:27.168555 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:02:27.168533 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa\": container with ID starting with 1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa not found: ID does not exist" containerID="1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa" Jul 16 18:02:27.168595 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.168562 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa"} err="failed to get container status \"1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa\": rpc error: code = NotFound desc = could not find container \"1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa\": container with ID starting with 1d74e5c3ee2c203d22ba74ab398b24e4dd4c5154fd3857aecbdfb050cb6147fa not found: ID does not exist" Jul 16 18:02:27.182416 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.182389 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p"] Jul 16 18:02:27.196446 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:27.196425 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5410de53-openshift-default-7896748dc9-kwz9p"] Jul 16 18:02:28.651928 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:02:28.651887 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" path="/var/lib/kubelet/pods/88bed580-4baf-4a62-8906-7b893df62ba5/volumes" Jul 16 18:04:13.306394 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.306361 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb"] Jul 16 18:04:13.306861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.306733 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9343db-ce76-4fb4-a6cc-aa81a458d83f" containerName="maas-api" Jul 16 18:04:13.306861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.306744 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9343db-ce76-4fb4-a6cc-aa81a458d83f" containerName="maas-api" Jul 16 18:04:13.306861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.306756 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" Jul 16 18:04:13.306861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.306763 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" Jul 16 18:04:13.306861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.306834 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" Jul 16 18:04:13.306861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.306845 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd9343db-ce76-4fb4-a6cc-aa81a458d83f" containerName="maas-api" Jul 16 18:04:13.309924 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.309907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.312949 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.312921 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2ef0185a-openshift-default-dockercfg-hrrk6\"" Jul 16 18:04:13.336574 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.336541 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb"] Jul 16 18:04:13.412461 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-data\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.412643 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412476 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-socket\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.412643 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412507 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-podinfo\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.412643 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-credential-socket\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.412643 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnw9h\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-kube-api-access-dnw9h\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.412643 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-certs\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.412643 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412624 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-envoy\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.412892 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istiod-ca-cert\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.412892 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.412744 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-token\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514329 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-data\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-socket\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-podinfo\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-credential-socket\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnw9h\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-kube-api-access-dnw9h\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-certs\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-envoy\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istiod-ca-cert\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514986 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-token\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514986 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514717 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-data\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514986 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-socket\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.514986 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.514909 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-certs\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.515194 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.515037 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-credential-socket\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.515498 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.515236 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istiod-ca-cert\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.516948 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.516922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-envoy\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.517048 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.517004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-podinfo\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.524419 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.524391 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnw9h\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-kube-api-access-dnw9h\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.524666 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.524648 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-token\") pod \"e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.620471 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.620370 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:13.773068 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.773037 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb"] Jul 16 18:04:13.774178 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:04:13.774154 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb167a54d_53f5_43ee_a6c8_c9fcfb8c19fb.slice/crio-726cec093dbb5612647b9d5b8a9a6d00a47739c4b39268d3ec0d05c3398ee623 WatchSource:0}: Error finding container 726cec093dbb5612647b9d5b8a9a6d00a47739c4b39268d3ec0d05c3398ee623: Status 404 returned error can't find the container with id 726cec093dbb5612647b9d5b8a9a6d00a47739c4b39268d3ec0d05c3398ee623 Jul 16 18:04:13.776305 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.776259 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:04:13.776394 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.776336 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:04:13.776394 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:13.776365 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:04:14.754361 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:14.754315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" event={"ID":"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb","Type":"ContainerStarted","Data":"d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b"} Jul 16 18:04:14.754731 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:14.754421 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" event={"ID":"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb","Type":"ContainerStarted","Data":"726cec093dbb5612647b9d5b8a9a6d00a47739c4b39268d3ec0d05c3398ee623"} Jul 16 18:04:14.779476 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:14.779414 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" podStartSLOduration=1.779391688 podStartE2EDuration="1.779391688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:04:14.776534349 +0000 UTC m=+2116.528173908" watchObservedRunningTime="2026-07-16 18:04:14.779391688 +0000 UTC m=+2116.531031247" Jul 16 18:04:15.620749 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:15.620710 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:15.625551 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:15.625526 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:15.760349 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:15.760318 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:15.761324 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:15.761307 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:28.847717 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:28.847684 2582 generic.go:356] "Generic (PLEG): container finished" podID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerID="d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b" exitCode=0 Jul 16 18:04:28.848135 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:28.847744 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" event={"ID":"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb","Type":"ContainerDied","Data":"d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b"} Jul 16 18:04:28.848183 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:28.848158 2582 scope.go:117] "RemoveContainer" containerID="d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b" Jul 16 18:04:28.849219 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:28.849193 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:04:28.849302 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:28.849262 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:04:28.849346 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:28.849319 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:04:29.858609 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:29.858578 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" event={"ID":"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb","Type":"ContainerStarted","Data":"a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa"} Jul 16 18:04:30.620824 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:30.620785 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:30.622096 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:30.622063 2582 patch_prober.go:28] interesting pod/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 16 18:04:30.622218 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:30.622122 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 16 18:04:30.760580 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:30.760542 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:31.621452 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:31.621412 2582 patch_prober.go:28] interesting pod/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 16 18:04:31.621836 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:31.621473 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 16 18:04:32.621322 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:32.621260 2582 patch_prober.go:28] interesting pod/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 16 18:04:32.621501 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:32.621362 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 16 18:04:33.621702 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:33.621663 2582 patch_prober.go:28] interesting pod/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 16 18:04:33.622098 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:33.621722 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 16 18:04:34.625195 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:34.625166 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:34.626166 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:34.626145 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:54.123707 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:54.123618 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb"] Jul 16 18:04:54.124263 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:54.123882 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" containerID="cri-o://a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa" gracePeriod=30 Jul 16 18:04:59.387265 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.387193 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:04:59.572515 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572477 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-certs\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.572515 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572525 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-data\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.572779 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572557 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-podinfo\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.572779 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-socket\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.572779 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572614 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istiod-ca-cert\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.572779 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnw9h\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-kube-api-access-dnw9h\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.572779 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572664 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-envoy\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.572779 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572751 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-credential-socket\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.572779 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572774 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-token\") pod \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\" (UID: \"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb\") " Jul 16 18:04:59.573147 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572812 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:04:59.573147 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.572942 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:04:59.573147 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.573027 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:04:59.573147 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.573081 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:04:59.573147 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.573108 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-credential-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:04:59.573147 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.573124 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:04:59.573147 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.573140 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-workload-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:04:59.573540 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.573406 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:04:59.575168 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.575138 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 18:04:59.575768 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.575732 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:04:59.575874 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.575804 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-kube-api-access-dnw9h" (OuterVolumeSpecName: "kube-api-access-dnw9h") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "kube-api-access-dnw9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:04:59.575874 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.575809 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" (UID: "b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:04:59.673948 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.673861 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istiod-ca-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:04:59.673948 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.673890 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnw9h\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-kube-api-access-dnw9h\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:04:59.673948 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.673902 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-envoy\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:04:59.673948 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.673910 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:04:59.673948 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.673918 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-data\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:04:59.673948 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:04:59.673925 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb-istio-podinfo\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:05:00.029690 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.029653 2582 generic.go:356] "Generic (PLEG): container finished" podID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerID="a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa" exitCode=0 Jul 16 18:05:00.029893 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.029707 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" Jul 16 18:05:00.029893 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.029745 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" event={"ID":"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb","Type":"ContainerDied","Data":"a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa"} Jul 16 18:05:00.029893 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.029814 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb" event={"ID":"b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb","Type":"ContainerDied","Data":"726cec093dbb5612647b9d5b8a9a6d00a47739c4b39268d3ec0d05c3398ee623"} Jul 16 18:05:00.029893 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.029846 2582 scope.go:117] "RemoveContainer" containerID="a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa" Jul 16 18:05:00.040203 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.040180 2582 scope.go:117] "RemoveContainer" containerID="d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b" Jul 16 18:05:00.049167 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.049149 2582 scope.go:117] "RemoveContainer" containerID="a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa" Jul 16 18:05:00.049482 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:05:00.049462 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa\": container with ID starting with a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa not found: ID does not exist" containerID="a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa" Jul 16 18:05:00.049537 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.049493 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa"} err="failed to get container status \"a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa\": rpc error: code = NotFound desc = could not find container \"a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa\": container with ID starting with a6b444bd766ea3a60401b16102f7d41c2d833fb477a9629bdd0d38c0bccc9daa not found: ID does not exist" Jul 16 18:05:00.049537 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.049512 2582 scope.go:117] "RemoveContainer" containerID="d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b" Jul 16 18:05:00.049802 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:05:00.049783 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b\": container with ID starting with d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b not found: ID does not exist" containerID="d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b" Jul 16 18:05:00.049859 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.049807 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b"} err="failed to get container status \"d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b\": rpc error: code = NotFound desc = could not find container \"d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b\": container with ID starting with d39508e801d0325b915d5a70e276c109029450019f27fe4338aa08ac9e3a047b not found: ID does not exist" Jul 16 18:05:00.057634 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.057605 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb"] Jul 16 18:05:00.063529 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.063501 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2ef0185a-openshift-default-54f448dd8c-vd2nb"] Jul 16 18:05:00.657648 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:00.657612 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" path="/var/lib/kubelet/pods/b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb/volumes" Jul 16 18:05:17.338012 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.337973 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8"] Jul 16 18:05:17.338439 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338424 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" Jul 16 18:05:17.338482 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338440 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" Jul 16 18:05:17.338482 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338453 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" Jul 16 18:05:17.338482 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338461 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" Jul 16 18:05:17.338482 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338475 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" Jul 16 18:05:17.338482 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338482 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" Jul 16 18:05:17.338653 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338561 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bed580-4baf-4a62-8906-7b893df62ba5" containerName="istio-proxy" Jul 16 18:05:17.338653 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338569 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" Jul 16 18:05:17.338653 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.338577 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b167a54d-53f5-43ee-a6c8-c9fcfb8c19fb" containerName="istio-proxy" Jul 16 18:05:17.341778 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.341758 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.344589 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.344570 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e4fca909-openshift-default-dockercfg-7dd5t\"" Jul 16 18:05:17.362426 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.362391 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8"] Jul 16 18:05:17.432910 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.432873 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-podinfo\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.433090 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.432960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-socket\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.433090 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.432993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-credential-socket\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.433090 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.433015 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-certs\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.433090 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.433031 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-envoy\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.433235 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.433160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgtgg\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-kube-api-access-sgtgg\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.433270 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.433239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-data\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.433270 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.433256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee4b0a-d570-4664-9423-d963b1e2502b-istiod-ca-cert\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.433371 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.433329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-token\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.534503 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-token\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.534503 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-podinfo\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.534774 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-socket\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.534774 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-credential-socket\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.534774 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-certs\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.534774 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534734 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-envoy\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.535011 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgtgg\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-kube-api-access-sgtgg\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.535011 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-data\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.535011 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee4b0a-d570-4664-9423-d963b1e2502b-istiod-ca-cert\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.535179 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.534975 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-socket\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.535179 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.535059 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-credential-socket\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.535319 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.535220 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-data\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.535319 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.535228 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-certs\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.535638 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.535619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee4b0a-d570-4664-9423-d963b1e2502b-istiod-ca-cert\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.537186 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.537166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-podinfo\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.537606 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.537588 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-envoy\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.546460 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.546427 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-token\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.546753 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.546733 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgtgg\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-kube-api-access-sgtgg\") pod \"e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.653229 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.653128 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:17.847394 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.847364 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8"] Jul 16 18:05:17.849342 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:05:17.849315 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ee4b0a_d570_4664_9423_d963b1e2502b.slice/crio-ad304c77f1156f3346af2ca50f005eedcfb36d8fbfdfd19abf809058ac103c21 WatchSource:0}: Error finding container ad304c77f1156f3346af2ca50f005eedcfb36d8fbfdfd19abf809058ac103c21: Status 404 returned error can't find the container with id ad304c77f1156f3346af2ca50f005eedcfb36d8fbfdfd19abf809058ac103c21 Jul 16 18:05:17.851499 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.851467 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:05:17.851593 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.851533 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:05:17.851593 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:17.851562 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:05:18.134484 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:18.134438 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" event={"ID":"75ee4b0a-d570-4664-9423-d963b1e2502b","Type":"ContainerStarted","Data":"87aad055e525088315f69657b998975111d2c1cae57b67552afd1a714ac5a2e1"} Jul 16 18:05:18.134693 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:18.134515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" event={"ID":"75ee4b0a-d570-4664-9423-d963b1e2502b","Type":"ContainerStarted","Data":"ad304c77f1156f3346af2ca50f005eedcfb36d8fbfdfd19abf809058ac103c21"} Jul 16 18:05:18.177599 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:18.177462 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" podStartSLOduration=1.177443062 podStartE2EDuration="1.177443062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:05:18.176421541 +0000 UTC m=+2179.928061134" watchObservedRunningTime="2026-07-16 18:05:18.177443062 +0000 UTC m=+2179.929082619" Jul 16 18:05:18.654502 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:18.654470 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:18.659631 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:18.659587 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:19.139609 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:19.139578 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:19.140456 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:19.140435 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:31.215086 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:31.214998 2582 generic.go:356] "Generic (PLEG): container finished" podID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerID="87aad055e525088315f69657b998975111d2c1cae57b67552afd1a714ac5a2e1" exitCode=0 Jul 16 18:05:31.215086 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:31.215065 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" event={"ID":"75ee4b0a-d570-4664-9423-d963b1e2502b","Type":"ContainerDied","Data":"87aad055e525088315f69657b998975111d2c1cae57b67552afd1a714ac5a2e1"} Jul 16 18:05:31.215539 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:31.215493 2582 scope.go:117] "RemoveContainer" containerID="87aad055e525088315f69657b998975111d2c1cae57b67552afd1a714ac5a2e1" Jul 16 18:05:31.216566 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:31.216538 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:05:31.216671 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:31.216606 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:05:31.216671 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:31.216638 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:05:32.222831 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:32.222794 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" event={"ID":"75ee4b0a-d570-4664-9423-d963b1e2502b","Type":"ContainerStarted","Data":"55ab1638a2de1b9400a9ac7b68412fdf434eaab258c7fa2e42cd8ea74289684b"} Jul 16 18:05:32.654329 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:32.654296 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:32.655594 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:32.655566 2582 patch_prober.go:28] interesting pod/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 16 18:05:32.655751 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:32.655640 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 16 18:05:33.654634 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:33.654595 2582 patch_prober.go:28] interesting pod/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 16 18:05:33.655024 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:33.654659 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 16 18:05:34.140492 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:34.140451 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:34.654829 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:34.654787 2582 patch_prober.go:28] interesting pod/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 16 18:05:34.655327 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:34.654846 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 16 18:05:35.654541 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:35.654498 2582 patch_prober.go:28] interesting pod/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 16 18:05:35.654732 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:35.654558 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 16 18:05:36.658717 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:36.658686 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:36.659954 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:36.659924 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:05:58.894350 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:05:58.894312 2582 scope.go:117] "RemoveContainer" containerID="8a9f6397cd27317225722fc00c692ba00b2c37a724cb53942b2f29cb37093bab" Jul 16 18:07:47.778887 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:47.778840 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8"] Jul 16 18:07:47.779455 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:47.779074 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" containerID="cri-o://55ab1638a2de1b9400a9ac7b68412fdf434eaab258c7fa2e42cd8ea74289684b" gracePeriod=30 Jul 16 18:07:51.662349 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:51.662248 2582 patch_prober.go:28] interesting pod/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 18:07:51.662771 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:51.662345 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:52.993451 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:52.993376 2582 generic.go:356] "Generic (PLEG): container finished" podID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerID="55ab1638a2de1b9400a9ac7b68412fdf434eaab258c7fa2e42cd8ea74289684b" exitCode=0 Jul 16 18:07:52.993451 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:52.993444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" event={"ID":"75ee4b0a-d570-4664-9423-d963b1e2502b","Type":"ContainerDied","Data":"55ab1638a2de1b9400a9ac7b68412fdf434eaab258c7fa2e42cd8ea74289684b"} Jul 16 18:07:52.993838 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:52.993507 2582 scope.go:117] "RemoveContainer" containerID="87aad055e525088315f69657b998975111d2c1cae57b67552afd1a714ac5a2e1" Jul 16 18:07:53.084780 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.084748 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:07:53.181374 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181343 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-credential-socket\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.181589 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181391 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee4b0a-d570-4664-9423-d963b1e2502b-istiod-ca-cert\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.181589 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181438 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-socket\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.181589 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181514 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-podinfo\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.181589 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181534 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-data\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.181589 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181566 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-token\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.181861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181598 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-certs\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.181861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181626 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgtgg\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-kube-api-access-sgtgg\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.181861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181622 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:07:53.181861 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181660 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-envoy\") pod \"75ee4b0a-d570-4664-9423-d963b1e2502b\" (UID: \"75ee4b0a-d570-4664-9423-d963b1e2502b\") " Jul 16 18:07:53.182065 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.181897 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ee4b0a-d570-4664-9423-d963b1e2502b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:07:53.182065 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.182001 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-credential-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:53.182065 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.182018 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee4b0a-d570-4664-9423-d963b1e2502b-istiod-ca-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:53.183371 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.182684 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:07:53.183371 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.183252 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-data" (OuterVolumeSpecName: "istio-data") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:07:53.183371 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.183342 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:07:53.184210 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.184184 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 18:07:53.184511 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.184474 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-kube-api-access-sgtgg" (OuterVolumeSpecName: "kube-api-access-sgtgg") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "kube-api-access-sgtgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:07:53.184612 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.184510 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:07:53.184706 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.184682 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-token" (OuterVolumeSpecName: "istio-token") pod "75ee4b0a-d570-4664-9423-d963b1e2502b" (UID: "75ee4b0a-d570-4664-9423-d963b1e2502b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:07:53.282580 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.282541 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:53.282580 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.282572 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-podinfo\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:53.282580 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.282582 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-data\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:53.282580 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.282590 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:53.282980 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.282598 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-workload-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:53.282980 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.282606 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgtgg\" (UniqueName: \"kubernetes.io/projected/75ee4b0a-d570-4664-9423-d963b1e2502b-kube-api-access-sgtgg\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:53.282980 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:53.282615 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee4b0a-d570-4664-9423-d963b1e2502b-istio-envoy\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:07:54.000944 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:54.000905 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" event={"ID":"75ee4b0a-d570-4664-9423-d963b1e2502b","Type":"ContainerDied","Data":"ad304c77f1156f3346af2ca50f005eedcfb36d8fbfdfd19abf809058ac103c21"} Jul 16 18:07:54.000944 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:54.000933 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8" Jul 16 18:07:54.001524 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:54.000965 2582 scope.go:117] "RemoveContainer" containerID="55ab1638a2de1b9400a9ac7b68412fdf434eaab258c7fa2e42cd8ea74289684b" Jul 16 18:07:54.030205 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:54.030173 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8"] Jul 16 18:07:54.037952 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:54.037924 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e4fca909-openshift-default-6f975fcbbc-5c9p8"] Jul 16 18:07:54.655985 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:07:54.655942 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" path="/var/lib/kubelet/pods/75ee4b0a-d570-4664-9423-d963b1e2502b/volumes" Jul 16 18:08:47.614879 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.614834 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj"] Jul 16 18:08:47.615413 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.615240 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" Jul 16 18:08:47.615413 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.615253 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" Jul 16 18:08:47.615413 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.615334 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" Jul 16 18:08:47.615413 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.615343 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" Jul 16 18:08:47.618386 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.618369 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.621535 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.621514 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1fa554-openshift-default-dockercfg-zr8gn\"" Jul 16 18:08:47.638359 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.638325 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj"] Jul 16 18:08:47.667840 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.667802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64802674-f8d6-493e-8989-ce50d9fd90c3-istiod-ca-cert\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.668037 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.667848 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwswc\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-kube-api-access-rwswc\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.668037 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.667938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-socket\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.668037 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.667968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-certs\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.668037 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.667993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-token\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.668261 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.668137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-envoy\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.668261 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.668174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-podinfo\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.668261 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.668221 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-credential-socket\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.668472 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.668262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-data\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.768866 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.768825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-envoy\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.768866 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.768864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-podinfo\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769133 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.768886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-credential-socket\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769133 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.768916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-data\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769133 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.769063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64802674-f8d6-493e-8989-ce50d9fd90c3-istiod-ca-cert\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769414 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.769390 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-credential-socket\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769414 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.769407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwswc\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-kube-api-access-rwswc\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769630 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.769468 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-data\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769630 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.769535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-socket\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769630 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.769568 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-certs\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769803 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.769644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-token\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.769994 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.769966 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64802674-f8d6-493e-8989-ce50d9fd90c3-istiod-ca-cert\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.770274 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.770248 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-socket\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.770407 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.770339 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-certs\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.771468 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.771448 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-envoy\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.771775 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.771757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-podinfo\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.782220 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.782190 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-token\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.782383 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.782315 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwswc\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-kube-api-access-rwswc\") pod \"e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:47.928404 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:47.928273 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:48.091060 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:48.091036 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj"] Jul 16 18:08:48.092375 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:08:48.092334 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64802674_f8d6_493e_8989_ce50d9fd90c3.slice/crio-851bbd5a88ba3a48f618e4c74a3e97bc7ae1ddfd06bafdf4a871c7f9f83b656e WatchSource:0}: Error finding container 851bbd5a88ba3a48f618e4c74a3e97bc7ae1ddfd06bafdf4a871c7f9f83b656e: Status 404 returned error can't find the container with id 851bbd5a88ba3a48f618e4c74a3e97bc7ae1ddfd06bafdf4a871c7f9f83b656e Jul 16 18:08:48.094423 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:48.094389 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:08:48.094513 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:48.094458 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:08:48.094513 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:48.094488 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:08:48.305435 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:48.305396 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" event={"ID":"64802674-f8d6-493e-8989-ce50d9fd90c3","Type":"ContainerStarted","Data":"f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3"} Jul 16 18:08:48.305631 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:48.305457 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" event={"ID":"64802674-f8d6-493e-8989-ce50d9fd90c3","Type":"ContainerStarted","Data":"851bbd5a88ba3a48f618e4c74a3e97bc7ae1ddfd06bafdf4a871c7f9f83b656e"} Jul 16 18:08:48.337849 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:48.337790 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" podStartSLOduration=1.337767621 podStartE2EDuration="1.337767621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:08:48.336715796 +0000 UTC m=+2390.088355358" watchObservedRunningTime="2026-07-16 18:08:48.337767621 +0000 UTC m=+2390.089407179" Jul 16 18:08:48.929100 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:48.929061 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:49.934013 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:49.933983 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:50.316312 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:50.316242 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:08:50.317397 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:08:50.317381 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:09:02.382583 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:02.382539 2582 generic.go:356] "Generic (PLEG): container finished" podID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerID="f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3" exitCode=0 Jul 16 18:09:02.383080 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:02.382600 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" event={"ID":"64802674-f8d6-493e-8989-ce50d9fd90c3","Type":"ContainerDied","Data":"f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3"} Jul 16 18:09:02.383160 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:02.383140 2582 scope.go:117] "RemoveContainer" containerID="f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3" Jul 16 18:09:02.384357 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:02.384330 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:09:02.384436 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:02.384408 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:09:02.384518 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:02.384465 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:09:03.394987 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:03.394937 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" event={"ID":"64802674-f8d6-493e-8989-ce50d9fd90c3","Type":"ContainerStarted","Data":"64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020"} Jul 16 18:09:03.928465 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:03.928421 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:09:03.929569 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:03.929544 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 16 18:09:03.929697 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:03.929596 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 16 18:09:04.928821 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:04.928784 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 16 18:09:04.929211 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:04.928838 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 16 18:09:05.316191 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:05.316146 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:09:05.928925 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:05.928888 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 16 18:09:05.929346 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:05.928960 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 16 18:09:06.929187 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:06.929148 2582 patch_prober.go:28] interesting pod/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 16 18:09:06.929597 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:06.929209 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 16 18:09:07.933489 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:07.933452 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:09:07.934532 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:07.934513 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:09:42.653550 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.653523 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd"] Jul 16 18:09:42.653971 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.653867 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" Jul 16 18:09:42.653971 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.653877 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ee4b0a-d570-4664-9423-d963b1e2502b" containerName="istio-proxy" Jul 16 18:09:42.657011 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.656993 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.660160 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.660134 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-36c084\"/\"kube-root-ca.crt\"" Jul 16 18:09:42.661354 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.661329 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-36c084\"/\"openshift-service-ca.crt\"" Jul 16 18:09:42.661495 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.661329 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-36c084\"/\"test-model-36c084-kserve-self-signed-certs\"" Jul 16 18:09:42.661495 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.661378 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-36c084\"/\"default-dockercfg-68q2d\"" Jul 16 18:09:42.663493 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.663467 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd"] Jul 16 18:09:42.788965 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.788914 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnznc\" (UniqueName: \"kubernetes.io/projected/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-kube-api-access-mnznc\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.789178 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.789080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-dshm\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.789178 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.789100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-model-cache\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.789332 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.789207 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tmp-dir\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.789332 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.789239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-home\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.789332 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.789303 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tls-certs\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890119 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tls-certs\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890333 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnznc\" (UniqueName: \"kubernetes.io/projected/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-kube-api-access-mnznc\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890333 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890203 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-dshm\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890333 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890217 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-model-cache\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890333 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tmp-dir\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890333 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-home\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890747 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890723 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-home\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890866 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-model-cache\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.890939 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.890901 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tmp-dir\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.892718 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.892699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-dshm\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.893180 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.893163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tls-certs\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.900382 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.900352 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnznc\" (UniqueName: \"kubernetes.io/projected/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-kube-api-access-mnznc\") pod \"test-model-36c084-kserve-66866559b5-8zdfd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:42.967337 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:42.967223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:43.116072 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:43.116047 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd"] Jul 16 18:09:43.117423 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:09:43.117397 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dcd72fb_a900_4976_89e5_5ddc4c02efbd.slice/crio-a41a4a3b4387476c1658a0cd84e016d0f7817bb88f24a32b04aeeb1395af9855 WatchSource:0}: Error finding container a41a4a3b4387476c1658a0cd84e016d0f7817bb88f24a32b04aeeb1395af9855: Status 404 returned error can't find the container with id a41a4a3b4387476c1658a0cd84e016d0f7817bb88f24a32b04aeeb1395af9855 Jul 16 18:09:43.624249 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:43.624210 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" event={"ID":"0dcd72fb-a900-4976-89e5-5ddc4c02efbd","Type":"ContainerStarted","Data":"466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7"} Jul 16 18:09:43.624249 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:43.624256 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" event={"ID":"0dcd72fb-a900-4976-89e5-5ddc4c02efbd","Type":"ContainerStarted","Data":"a41a4a3b4387476c1658a0cd84e016d0f7817bb88f24a32b04aeeb1395af9855"} Jul 16 18:09:43.645232 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:43.645181 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" podStartSLOduration=1.645165482 podStartE2EDuration="1.645165482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:09:43.643696033 +0000 UTC m=+2445.395335591" watchObservedRunningTime="2026-07-16 18:09:43.645165482 +0000 UTC m=+2445.396805040" Jul 16 18:09:52.968357 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:52.968320 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:52.968818 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:52.968396 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:52.981415 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:52.981378 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:09:53.692886 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:09:53.692844 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:10:05.755210 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:05.755171 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd"] Jul 16 18:10:20.776058 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:20.776006 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" podUID="0dcd72fb-a900-4976-89e5-5ddc4c02efbd" containerName="main" containerID="cri-o://466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7" gracePeriod=45 Jul 16 18:10:21.042806 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.042780 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:10:21.153241 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153206 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnznc\" (UniqueName: \"kubernetes.io/projected/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-kube-api-access-mnznc\") pod \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " Jul 16 18:10:21.153241 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153247 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-dshm\") pod \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " Jul 16 18:10:21.153525 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153264 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tmp-dir\") pod \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " Jul 16 18:10:21.153525 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153321 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-model-cache\") pod \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " Jul 16 18:10:21.153525 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153355 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tls-certs\") pod \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " Jul 16 18:10:21.153525 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153406 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-home\") pod \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\" (UID: \"0dcd72fb-a900-4976-89e5-5ddc4c02efbd\") " Jul 16 18:10:21.153786 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153594 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0dcd72fb-a900-4976-89e5-5ddc4c02efbd" (UID: "0dcd72fb-a900-4976-89e5-5ddc4c02efbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:21.153786 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153644 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-model-cache" (OuterVolumeSpecName: "model-cache") pod "0dcd72fb-a900-4976-89e5-5ddc4c02efbd" (UID: "0dcd72fb-a900-4976-89e5-5ddc4c02efbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:21.153786 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153706 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-home" (OuterVolumeSpecName: "home") pod "0dcd72fb-a900-4976-89e5-5ddc4c02efbd" (UID: "0dcd72fb-a900-4976-89e5-5ddc4c02efbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:21.153898 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153856 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-model-cache\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:21.153898 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153874 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-home\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:21.153898 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.153888 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tmp-dir\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:21.155739 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.155706 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0dcd72fb-a900-4976-89e5-5ddc4c02efbd" (UID: "0dcd72fb-a900-4976-89e5-5ddc4c02efbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:10:21.155860 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.155742 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-dshm" (OuterVolumeSpecName: "dshm") pod "0dcd72fb-a900-4976-89e5-5ddc4c02efbd" (UID: "0dcd72fb-a900-4976-89e5-5ddc4c02efbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:21.155860 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.155771 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-kube-api-access-mnznc" (OuterVolumeSpecName: "kube-api-access-mnznc") pod "0dcd72fb-a900-4976-89e5-5ddc4c02efbd" (UID: "0dcd72fb-a900-4976-89e5-5ddc4c02efbd"). InnerVolumeSpecName "kube-api-access-mnznc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:10:21.254368 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.254340 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-tls-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:21.254368 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.254366 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnznc\" (UniqueName: \"kubernetes.io/projected/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-kube-api-access-mnznc\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:21.254368 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.254376 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd72fb-a900-4976-89e5-5ddc4c02efbd-dshm\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:21.841047 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.841010 2582 generic.go:356] "Generic (PLEG): container finished" podID="0dcd72fb-a900-4976-89e5-5ddc4c02efbd" containerID="466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7" exitCode=0 Jul 16 18:10:21.841548 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.841054 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" Jul 16 18:10:21.841548 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.841097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" event={"ID":"0dcd72fb-a900-4976-89e5-5ddc4c02efbd","Type":"ContainerDied","Data":"466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7"} Jul 16 18:10:21.841548 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.841140 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd" event={"ID":"0dcd72fb-a900-4976-89e5-5ddc4c02efbd","Type":"ContainerDied","Data":"a41a4a3b4387476c1658a0cd84e016d0f7817bb88f24a32b04aeeb1395af9855"} Jul 16 18:10:21.841548 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.841160 2582 scope.go:117] "RemoveContainer" containerID="466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7" Jul 16 18:10:21.853350 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.853304 2582 scope.go:117] "RemoveContainer" containerID="466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7" Jul 16 18:10:21.853640 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:10:21.853611 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7\": container with ID starting with 466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7 not found: ID does not exist" containerID="466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7" Jul 16 18:10:21.853721 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.853655 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7"} err="failed to get container status \"466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7\": rpc error: code = NotFound desc = could not find container \"466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7\": container with ID starting with 466ced399378b387664e7b5a44a22f0d75247c6c7369edae71ed4288eac0ebf7 not found: ID does not exist" Jul 16 18:10:21.869218 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.869184 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd"] Jul 16 18:10:21.874166 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:21.874142 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-36c084/test-model-36c084-kserve-66866559b5-8zdfd"] Jul 16 18:10:22.652453 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:22.652419 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dcd72fb-a900-4976-89e5-5ddc4c02efbd" path="/var/lib/kubelet/pods/0dcd72fb-a900-4976-89e5-5ddc4c02efbd/volumes" Jul 16 18:10:28.209427 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:28.209395 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj"] Jul 16 18:10:28.209903 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:28.209622 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" containerID="cri-o://64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020" gracePeriod=30 Jul 16 18:10:33.476710 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.476637 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:10:33.569506 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569464 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-podinfo\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.569698 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569516 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-token\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.569698 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569561 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-data\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.569824 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569703 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-socket\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.569824 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569771 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwswc\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-kube-api-access-rwswc\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.569921 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569835 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-credential-socket\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.569921 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569890 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64802674-f8d6-493e-8989-ce50d9fd90c3-istiod-ca-cert\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.570023 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-certs\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.570023 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569970 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-envoy\") pod \"64802674-f8d6-493e-8989-ce50d9fd90c3\" (UID: \"64802674-f8d6-493e-8989-ce50d9fd90c3\") " Jul 16 18:10:33.570355 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.569963 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:33.570355 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.570192 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:33.570355 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.570216 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:33.570874 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.570595 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:33.570874 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.570689 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64802674-f8d6-493e-8989-ce50d9fd90c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:10:33.572012 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.571982 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:10:33.572012 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.572004 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 18:10:33.572202 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.572181 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:10:33.572269 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.572233 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-kube-api-access-rwswc" (OuterVolumeSpecName: "kube-api-access-rwswc") pod "64802674-f8d6-493e-8989-ce50d9fd90c3" (UID: "64802674-f8d6-493e-8989-ce50d9fd90c3"). InnerVolumeSpecName "kube-api-access-rwswc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:10:33.671733 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671675 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.671733 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671723 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwswc\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-kube-api-access-rwswc\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.671733 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671736 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-credential-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.671733 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671748 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64802674-f8d6-493e-8989-ce50d9fd90c3-istiod-ca-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.672032 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671762 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-workload-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.672032 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671772 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-envoy\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.672032 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671783 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-podinfo\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.672032 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671793 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.672032 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.671804 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64802674-f8d6-493e-8989-ce50d9fd90c3-istio-data\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:10:33.915930 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.915892 2582 generic.go:356] "Generic (PLEG): container finished" podID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerID="64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020" exitCode=0 Jul 16 18:10:33.916142 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.915946 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" Jul 16 18:10:33.916142 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.915978 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" event={"ID":"64802674-f8d6-493e-8989-ce50d9fd90c3","Type":"ContainerDied","Data":"64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020"} Jul 16 18:10:33.916142 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.916035 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj" event={"ID":"64802674-f8d6-493e-8989-ce50d9fd90c3","Type":"ContainerDied","Data":"851bbd5a88ba3a48f618e4c74a3e97bc7ae1ddfd06bafdf4a871c7f9f83b656e"} Jul 16 18:10:33.916142 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.916060 2582 scope.go:117] "RemoveContainer" containerID="64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020" Jul 16 18:10:33.926069 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.926051 2582 scope.go:117] "RemoveContainer" containerID="f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3" Jul 16 18:10:33.934653 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.934636 2582 scope.go:117] "RemoveContainer" containerID="64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020" Jul 16 18:10:33.934905 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:10:33.934887 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020\": container with ID starting with 64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020 not found: ID does not exist" containerID="64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020" Jul 16 18:10:33.934969 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.934914 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020"} err="failed to get container status \"64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020\": rpc error: code = NotFound desc = could not find container \"64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020\": container with ID starting with 64a97423d5d2dc555b67fe1b84e2481951ad90255cec5b2a457c19feb1ba4020 not found: ID does not exist" Jul 16 18:10:33.934969 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.934932 2582 scope.go:117] "RemoveContainer" containerID="f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3" Jul 16 18:10:33.935161 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:10:33.935145 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3\": container with ID starting with f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3 not found: ID does not exist" containerID="f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3" Jul 16 18:10:33.935201 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.935166 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3"} err="failed to get container status \"f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3\": rpc error: code = NotFound desc = could not find container \"f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3\": container with ID starting with f98d89d6b5c94082ed0440f8ced1545e0e120d25857fa36ef02dec2d889321b3 not found: ID does not exist" Jul 16 18:10:33.943891 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.943868 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj"] Jul 16 18:10:33.951449 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:33.951431 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1fa554-openshift-default-5587796f69-jlsrj"] Jul 16 18:10:34.652437 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:34.652406 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" path="/var/lib/kubelet/pods/64802674-f8d6-493e-8989-ce50d9fd90c3/volumes" Jul 16 18:10:53.237434 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.237350 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq"] Jul 16 18:10:53.237928 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.237908 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" Jul 16 18:10:53.237991 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.237932 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" Jul 16 18:10:53.237991 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.237950 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" Jul 16 18:10:53.237991 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.237958 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" Jul 16 18:10:53.238131 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.237993 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcd72fb-a900-4976-89e5-5ddc4c02efbd" containerName="main" Jul 16 18:10:53.238131 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.238001 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcd72fb-a900-4976-89e5-5ddc4c02efbd" containerName="main" Jul 16 18:10:53.238131 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.238093 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" Jul 16 18:10:53.238131 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.238109 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64802674-f8d6-493e-8989-ce50d9fd90c3" containerName="istio-proxy" Jul 16 18:10:53.238131 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.238119 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcd72fb-a900-4976-89e5-5ddc4c02efbd" containerName="main" Jul 16 18:10:53.241514 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.241492 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.244378 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.244358 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0baa98-openshift-default-dockercfg-czqdb\"" Jul 16 18:10:53.253371 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.253351 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq"] Jul 16 18:10:53.340438 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-certs\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.340438 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-podinfo\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.340692 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-token\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.340692 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istiod-ca-cert\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.340692 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340624 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-envoy\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.340692 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340644 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-data\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.340692 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-socket\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.340888 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340707 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-credential-socket\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.340888 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.340748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh24l\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-kube-api-access-rh24l\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441388 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-token\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441593 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istiod-ca-cert\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441593 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-envoy\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441593 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-data\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441593 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-socket\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441593 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-credential-socket\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441593 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441568 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh24l\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-kube-api-access-rh24l\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441908 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-certs\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.441908 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-podinfo\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.442018 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.441939 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-data\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.442077 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.442057 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-socket\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.442153 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.442122 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-credential-socket\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.442211 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.442194 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-certs\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.442555 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.442407 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istiod-ca-cert\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.444076 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.444058 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-envoy\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.444326 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.444306 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-podinfo\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.451096 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.451069 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-token\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.451482 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.451462 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh24l\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-kube-api-access-rh24l\") pod \"e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.551940 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.551904 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:53.704440 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.704412 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq"] Jul 16 18:10:53.706264 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:10:53.706233 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed8ccb4_8491_4fd3_abb4_48dc2ad3f83a.slice/crio-11f25c2eb149551a56ef014f13ca7babef999957277e583370953a5380d6e145 WatchSource:0}: Error finding container 11f25c2eb149551a56ef014f13ca7babef999957277e583370953a5380d6e145: Status 404 returned error can't find the container with id 11f25c2eb149551a56ef014f13ca7babef999957277e583370953a5380d6e145 Jul 16 18:10:53.708953 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.708920 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:10:53.709036 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.708985 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:10:53.709036 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:53.709012 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:10:54.036466 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:54.036427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" event={"ID":"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a","Type":"ContainerStarted","Data":"0fb73636ef0e03bd75591c7fe852bb41bf6ec938e91904deef4ad2547b272e81"} Jul 16 18:10:54.036650 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:54.036489 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" event={"ID":"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a","Type":"ContainerStarted","Data":"11f25c2eb149551a56ef014f13ca7babef999957277e583370953a5380d6e145"} Jul 16 18:10:54.061880 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:54.061818 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" podStartSLOduration=1.06180229 podStartE2EDuration="1.06180229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:10:54.059084951 +0000 UTC m=+2515.810724535" watchObservedRunningTime="2026-07-16 18:10:54.06180229 +0000 UTC m=+2515.813441848" Jul 16 18:10:54.552884 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:54.552850 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:54.557814 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:54.557790 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:55.042135 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:55.042104 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:10:55.043146 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:10:55.043128 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:11:13.146167 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:13.146135 2582 generic.go:356] "Generic (PLEG): container finished" podID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerID="0fb73636ef0e03bd75591c7fe852bb41bf6ec938e91904deef4ad2547b272e81" exitCode=0 Jul 16 18:11:13.146629 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:13.146211 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" event={"ID":"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a","Type":"ContainerDied","Data":"0fb73636ef0e03bd75591c7fe852bb41bf6ec938e91904deef4ad2547b272e81"} Jul 16 18:11:13.146696 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:13.146677 2582 scope.go:117] "RemoveContainer" containerID="0fb73636ef0e03bd75591c7fe852bb41bf6ec938e91904deef4ad2547b272e81" Jul 16 18:11:13.147661 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:13.147635 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:11:13.147713 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:13.147699 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:11:13.147747 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:13.147734 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 18:11:14.154082 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:14.154047 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" event={"ID":"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a","Type":"ContainerStarted","Data":"ba5de78a216c7885217444d359d96af9aba2bd0eb9313c20e77843ed2d1111c0"} Jul 16 18:11:14.552500 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:14.552467 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:11:14.552726 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:14.552705 2582 patch_prober.go:28] interesting pod/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 16 18:11:14.552781 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:14.552754 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 16 18:11:15.552806 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:15.552764 2582 patch_prober.go:28] interesting pod/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 16 18:11:15.553186 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:15.552822 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 16 18:11:16.553307 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:16.553251 2582 patch_prober.go:28] interesting pod/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 16 18:11:16.553704 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:16.553335 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 16 18:11:17.552615 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:17.552580 2582 patch_prober.go:28] interesting pod/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 16 18:11:17.552797 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:17.552653 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 16 18:11:18.556313 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:18.556250 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:11:18.556772 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:18.556752 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:11:18.557489 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:18.557466 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:11:45.186669 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:45.186634 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq"] Jul 16 18:11:45.187256 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:45.186907 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" containerID="cri-o://ba5de78a216c7885217444d359d96af9aba2bd0eb9313c20e77843ed2d1111c0" gracePeriod=30 Jul 16 18:11:48.559738 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:48.559701 2582 patch_prober.go:28] interesting pod/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 18:11:48.560118 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:48.559764 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:11:50.354518 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.354484 2582 generic.go:356] "Generic (PLEG): container finished" podID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerID="ba5de78a216c7885217444d359d96af9aba2bd0eb9313c20e77843ed2d1111c0" exitCode=0 Jul 16 18:11:50.354919 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.354558 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" event={"ID":"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a","Type":"ContainerDied","Data":"ba5de78a216c7885217444d359d96af9aba2bd0eb9313c20e77843ed2d1111c0"} Jul 16 18:11:50.354919 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.354620 2582 scope.go:117] "RemoveContainer" containerID="0fb73636ef0e03bd75591c7fe852bb41bf6ec938e91904deef4ad2547b272e81" Jul 16 18:11:50.443821 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.443799 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:11:50.604452 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604409 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-socket\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604650 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604475 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-certs\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604650 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604512 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh24l\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-kube-api-access-rh24l\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604650 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604538 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-podinfo\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604650 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604559 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-token\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604650 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604610 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-credential-socket\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604869 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604676 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-data\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604869 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604715 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-envoy\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604869 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604756 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istiod-ca-cert\") pod \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\" (UID: \"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a\") " Jul 16 18:11:50.604869 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604769 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:11:50.605069 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604881 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:11:50.605069 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.604992 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:11:50.605069 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.605065 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:50.605230 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.605080 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-workload-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:50.605230 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.605088 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-credential-socket\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:50.605496 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.605407 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:11:50.605674 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.605631 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:11:50.606953 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.606931 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:11:50.607186 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.607158 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:11:50.607433 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.607410 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-kube-api-access-rh24l" (OuterVolumeSpecName: "kube-api-access-rh24l") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "kube-api-access-rh24l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:11:50.607547 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.607519 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" (UID: "9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 18:11:50.706461 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.706429 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh24l\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-kube-api-access-rh24l\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:50.706461 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.706457 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-podinfo\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:50.706461 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.706470 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-token\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:50.706686 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.706483 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-data\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:50.706686 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.706491 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istio-envoy\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:50.706686 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:50.706498 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a-istiod-ca-cert\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:11:51.361092 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:51.361060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" event={"ID":"9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a","Type":"ContainerDied","Data":"11f25c2eb149551a56ef014f13ca7babef999957277e583370953a5380d6e145"} Jul 16 18:11:51.361092 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:51.361094 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq" Jul 16 18:11:51.361578 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:51.361118 2582 scope.go:117] "RemoveContainer" containerID="ba5de78a216c7885217444d359d96af9aba2bd0eb9313c20e77843ed2d1111c0" Jul 16 18:11:51.383807 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:51.383777 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq"] Jul 16 18:11:51.389220 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:51.389195 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0baa98-openshift-default-64d4c8dfbf-2l7mq"] Jul 16 18:11:52.651944 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:11:52.651910 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" path="/var/lib/kubelet/pods/9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a/volumes" Jul 16 18:12:17.263212 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.263185 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx"] Jul 16 18:12:17.263680 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.263579 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" Jul 16 18:12:17.263680 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.263591 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" Jul 16 18:12:17.263680 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.263607 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" Jul 16 18:12:17.263680 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.263612 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" Jul 16 18:12:17.263680 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.263674 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" Jul 16 18:12:17.263680 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.263681 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed8ccb4-8491-4fd3-abb4-48dc2ad3f83a" containerName="istio-proxy" Jul 16 18:12:17.266651 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.266623 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.270531 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.270511 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nsr52\"" Jul 16 18:12:17.271048 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.271026 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7c2a35-serving-cert\"" Jul 16 18:12:17.286057 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.286027 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx"] Jul 16 18:12:17.332175 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.332147 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrbp\" (UniqueName: \"kubernetes.io/projected/f4b683c8-cc87-4315-97cb-11785b57b22b-kube-api-access-tcrbp\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.332396 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.332182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.332396 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.332255 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b683c8-cc87-4315-97cb-11785b57b22b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.433458 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.433416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b683c8-cc87-4315-97cb-11785b57b22b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.433668 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.433516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrbp\" (UniqueName: \"kubernetes.io/projected/f4b683c8-cc87-4315-97cb-11785b57b22b-kube-api-access-tcrbp\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.433668 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.433547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.433799 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:12:17.433681 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-serving-cert: secret "maas-api-e2e-shared-a-7c2a35-serving-cert" not found Jul 16 18:12:17.433799 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:12:17.433749 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls podName:f4b683c8-cc87-4315-97cb-11785b57b22b nodeName:}" failed. No retries permitted until 2026-07-16 18:12:17.93372893 +0000 UTC m=+2599.685368477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls") pod "maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" (UID: "f4b683c8-cc87-4315-97cb-11785b57b22b") : secret "maas-api-e2e-shared-a-7c2a35-serving-cert" not found Jul 16 18:12:17.434061 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.434041 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b683c8-cc87-4315-97cb-11785b57b22b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.456170 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.456139 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrbp\" (UniqueName: \"kubernetes.io/projected/f4b683c8-cc87-4315-97cb-11785b57b22b-kube-api-access-tcrbp\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.936704 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.936667 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:17.939218 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:17.939195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls\") pod \"maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:18.177111 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:18.177080 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:18.327072 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:18.327044 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx"] Jul 16 18:12:18.328462 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:12:18.328436 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b683c8_cc87_4315_97cb_11785b57b22b.slice/crio-b83103d74ba9c0a2697ba69dd12d8cd293ab9c1009eaa9040203215ab601b661 WatchSource:0}: Error finding container b83103d74ba9c0a2697ba69dd12d8cd293ab9c1009eaa9040203215ab601b661: Status 404 returned error can't find the container with id b83103d74ba9c0a2697ba69dd12d8cd293ab9c1009eaa9040203215ab601b661 Jul 16 18:12:18.329833 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:18.329799 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 18:12:18.508570 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:18.508478 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" event={"ID":"f4b683c8-cc87-4315-97cb-11785b57b22b","Type":"ContainerStarted","Data":"b83103d74ba9c0a2697ba69dd12d8cd293ab9c1009eaa9040203215ab601b661"} Jul 16 18:12:19.515501 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:19.515414 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" event={"ID":"f4b683c8-cc87-4315-97cb-11785b57b22b","Type":"ContainerStarted","Data":"be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f"} Jul 16 18:12:19.515501 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:19.515477 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:19.542737 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:19.542686 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" podStartSLOduration=2.142000435 podStartE2EDuration="2.542670218s" totalImagesPullingTime="400.669783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:12:19.54038733 +0000 UTC m=+2601.292026897" watchObservedRunningTime="2026-07-16 18:12:19.542670218 +0000 UTC m=+2601.294309801" Jul 16 18:12:25.252638 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.252608 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4"] Jul 16 18:12:25.256547 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.256524 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.262098 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.262077 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d3e547-serving-cert\"" Jul 16 18:12:25.271436 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.271405 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4"] Jul 16 18:12:25.302822 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.302776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.302995 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.302832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twsb\" (UniqueName: \"kubernetes.io/projected/c92060f3-1e7e-4082-bf85-52484f6e1586-kube-api-access-8twsb\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.302995 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.302916 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92060f3-1e7e-4082-bf85-52484f6e1586-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.404184 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.404145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.404413 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.404193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twsb\" (UniqueName: \"kubernetes.io/projected/c92060f3-1e7e-4082-bf85-52484f6e1586-kube-api-access-8twsb\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.404413 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.404217 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92060f3-1e7e-4082-bf85-52484f6e1586-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.404413 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:12:25.404315 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-serving-cert: secret "maas-api-e2e-shared-b-d3e547-serving-cert" not found Jul 16 18:12:25.404413 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:12:25.404388 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls podName:c92060f3-1e7e-4082-bf85-52484f6e1586 nodeName:}" failed. No retries permitted until 2026-07-16 18:12:25.904369462 +0000 UTC m=+2607.656009014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls") pod "maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" (UID: "c92060f3-1e7e-4082-bf85-52484f6e1586") : secret "maas-api-e2e-shared-b-d3e547-serving-cert" not found Jul 16 18:12:25.404869 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.404847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92060f3-1e7e-4082-bf85-52484f6e1586-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.420569 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.420535 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twsb\" (UniqueName: \"kubernetes.io/projected/c92060f3-1e7e-4082-bf85-52484f6e1586-kube-api-access-8twsb\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.526156 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.526086 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:12:25.911155 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.911120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:25.913694 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:25.913666 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls\") pod \"maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:26.167209 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:26.167121 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:26.325502 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:26.325479 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4"] Jul 16 18:12:26.327171 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:12:26.327142 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92060f3_1e7e_4082_bf85_52484f6e1586.slice/crio-407a500b251c673ef32b05ef6daaa7a5919835ef8a7cc0ed43c47fde86d42d40 WatchSource:0}: Error finding container 407a500b251c673ef32b05ef6daaa7a5919835ef8a7cc0ed43c47fde86d42d40: Status 404 returned error can't find the container with id 407a500b251c673ef32b05ef6daaa7a5919835ef8a7cc0ed43c47fde86d42d40 Jul 16 18:12:26.554297 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:26.554257 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" event={"ID":"c92060f3-1e7e-4082-bf85-52484f6e1586","Type":"ContainerStarted","Data":"407a500b251c673ef32b05ef6daaa7a5919835ef8a7cc0ed43c47fde86d42d40"} Jul 16 18:12:27.561364 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:27.561320 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" event={"ID":"c92060f3-1e7e-4082-bf85-52484f6e1586","Type":"ContainerStarted","Data":"fb398568b6a3b46a592e36b40fa0b7a2a50cb9847ca36f26d3a5f74254f9a908"} Jul 16 18:12:27.561766 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:27.561509 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:33.572421 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:33.572393 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:12:33.592541 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:12:33.592400 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" podStartSLOduration=8.251551286 podStartE2EDuration="8.592358526s" totalImagesPullingTime="340.80724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:12:27.591912408 +0000 UTC m=+2609.343551966" watchObservedRunningTime="2026-07-16 18:12:33.592358526 +0000 UTC m=+2615.343998085" Jul 16 18:15:40.850474 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:40.850436 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7"] Jul 16 18:15:40.854026 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:40.854009 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:40.858686 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:40.858649 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7c2a35\"/\"openshift-service-ca.crt\"" Jul 16 18:15:40.858686 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:40.858680 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7c2a35\"/\"default-dockercfg-hsk5x\"" Jul 16 18:15:40.858864 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:40.858703 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7c2a35\"/\"sub-test-model-7c2a35-kserve-self-signed-certs\"" Jul 16 18:15:40.858864 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:40.858733 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7c2a35\"/\"kube-root-ca.crt\"" Jul 16 18:15:40.864352 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:40.864325 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7"] Jul 16 18:15:41.023599 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.023566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-dshm\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.023805 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.023620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-model-cache\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.023805 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.023732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-home\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.023805 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.023785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-tmp-dir\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.023993 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.023841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5cd2dc-4977-4423-b094-634218d51fca-tls-certs\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.023993 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.023906 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vz6\" (UniqueName: \"kubernetes.io/projected/0d5cd2dc-4977-4423-b094-634218d51fca-kube-api-access-b6vz6\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.124727 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.124632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-model-cache\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.124727 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.124684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-home\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.124951 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.124804 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-tmp-dir\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.124951 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.124856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5cd2dc-4977-4423-b094-634218d51fca-tls-certs\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.124951 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.124920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vz6\" (UniqueName: \"kubernetes.io/projected/0d5cd2dc-4977-4423-b094-634218d51fca-kube-api-access-b6vz6\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.125117 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.124972 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-dshm\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.125117 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.125100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-model-cache\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.125230 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.125132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-home\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.125230 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.125204 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-tmp-dir\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.127315 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.127269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-dshm\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.127604 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.127586 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5cd2dc-4977-4423-b094-634218d51fca-tls-certs\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.135655 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.135627 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vz6\" (UniqueName: \"kubernetes.io/projected/0d5cd2dc-4977-4423-b094-634218d51fca-kube-api-access-b6vz6\") pod \"sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.164501 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.164467 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:41.312865 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.312830 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7"] Jul 16 18:15:41.314578 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:15:41.314555 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d5cd2dc_4977_4423_b094_634218d51fca.slice/crio-aa23b5a7f9e0a701bfc44f48f8980c7562ab37eb8d31067cc5f95ec68ee5a565 WatchSource:0}: Error finding container aa23b5a7f9e0a701bfc44f48f8980c7562ab37eb8d31067cc5f95ec68ee5a565: Status 404 returned error can't find the container with id aa23b5a7f9e0a701bfc44f48f8980c7562ab37eb8d31067cc5f95ec68ee5a565 Jul 16 18:15:41.651760 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.651657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" event={"ID":"0d5cd2dc-4977-4423-b094-634218d51fca","Type":"ContainerStarted","Data":"83e0c964c16f9dd7a624ff4f6280db9723996517bf1546c9c5fc421bc4bd47b9"} Jul 16 18:15:41.651760 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.651701 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" event={"ID":"0d5cd2dc-4977-4423-b094-634218d51fca","Type":"ContainerStarted","Data":"aa23b5a7f9e0a701bfc44f48f8980c7562ab37eb8d31067cc5f95ec68ee5a565"} Jul 16 18:15:41.672844 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:41.672789 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" podStartSLOduration=1.672772001 podStartE2EDuration="1.672772001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:15:41.670628162 +0000 UTC m=+2803.422267720" watchObservedRunningTime="2026-07-16 18:15:41.672772001 +0000 UTC m=+2803.424411558" Jul 16 18:15:51.164832 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:51.164785 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:51.164832 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:51.164846 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:51.177581 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:51.177555 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:15:51.720754 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:15:51.720726 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:16:52.999391 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:16:52.999273 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7"] Jul 16 18:17:08.021898 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.021846 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" podUID="0d5cd2dc-4977-4423-b094-634218d51fca" containerName="main" containerID="cri-o://83e0c964c16f9dd7a624ff4f6280db9723996517bf1546c9c5fc421bc4bd47b9" gracePeriod=45 Jul 16 18:17:08.158605 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.158570 2582 generic.go:356] "Generic (PLEG): container finished" podID="0d5cd2dc-4977-4423-b094-634218d51fca" containerID="83e0c964c16f9dd7a624ff4f6280db9723996517bf1546c9c5fc421bc4bd47b9" exitCode=0 Jul 16 18:17:08.158786 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.158627 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" event={"ID":"0d5cd2dc-4977-4423-b094-634218d51fca","Type":"ContainerDied","Data":"83e0c964c16f9dd7a624ff4f6280db9723996517bf1546c9c5fc421bc4bd47b9"} Jul 16 18:17:08.302196 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.302165 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:17:08.422808 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.422772 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-model-cache\") pod \"0d5cd2dc-4977-4423-b094-634218d51fca\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " Jul 16 18:17:08.423001 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.422824 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6vz6\" (UniqueName: \"kubernetes.io/projected/0d5cd2dc-4977-4423-b094-634218d51fca-kube-api-access-b6vz6\") pod \"0d5cd2dc-4977-4423-b094-634218d51fca\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " Jul 16 18:17:08.423001 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.422862 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-tmp-dir\") pod \"0d5cd2dc-4977-4423-b094-634218d51fca\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " Jul 16 18:17:08.423001 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.422880 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-dshm\") pod \"0d5cd2dc-4977-4423-b094-634218d51fca\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " Jul 16 18:17:08.423001 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.422906 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-home\") pod \"0d5cd2dc-4977-4423-b094-634218d51fca\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " Jul 16 18:17:08.423001 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.422963 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5cd2dc-4977-4423-b094-634218d51fca-tls-certs\") pod \"0d5cd2dc-4977-4423-b094-634218d51fca\" (UID: \"0d5cd2dc-4977-4423-b094-634218d51fca\") " Jul 16 18:17:08.423373 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.423065 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d5cd2dc-4977-4423-b094-634218d51fca" (UID: "0d5cd2dc-4977-4423-b094-634218d51fca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:17:08.423373 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.423211 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d5cd2dc-4977-4423-b094-634218d51fca" (UID: "0d5cd2dc-4977-4423-b094-634218d51fca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:17:08.423373 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.423223 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-home" (OuterVolumeSpecName: "home") pod "0d5cd2dc-4977-4423-b094-634218d51fca" (UID: "0d5cd2dc-4977-4423-b094-634218d51fca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:17:08.423527 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.423372 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-home\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:17:08.423527 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.423394 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-model-cache\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:17:08.423527 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.423430 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-tmp-dir\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:17:08.425330 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.425305 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5cd2dc-4977-4423-b094-634218d51fca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d5cd2dc-4977-4423-b094-634218d51fca" (UID: "0d5cd2dc-4977-4423-b094-634218d51fca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:17:08.425431 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.425366 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5cd2dc-4977-4423-b094-634218d51fca-kube-api-access-b6vz6" (OuterVolumeSpecName: "kube-api-access-b6vz6") pod "0d5cd2dc-4977-4423-b094-634218d51fca" (UID: "0d5cd2dc-4977-4423-b094-634218d51fca"). InnerVolumeSpecName "kube-api-access-b6vz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:17:08.425431 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.425388 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-dshm" (OuterVolumeSpecName: "dshm") pod "0d5cd2dc-4977-4423-b094-634218d51fca" (UID: "0d5cd2dc-4977-4423-b094-634218d51fca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 18:17:08.524740 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.524648 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5cd2dc-4977-4423-b094-634218d51fca-dshm\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:17:08.524740 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.524681 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5cd2dc-4977-4423-b094-634218d51fca-tls-certs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:17:08.524740 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:08.524695 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6vz6\" (UniqueName: \"kubernetes.io/projected/0d5cd2dc-4977-4423-b094-634218d51fca-kube-api-access-b6vz6\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:17:09.165530 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:09.165499 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" Jul 16 18:17:09.165949 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:09.165538 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7" event={"ID":"0d5cd2dc-4977-4423-b094-634218d51fca","Type":"ContainerDied","Data":"aa23b5a7f9e0a701bfc44f48f8980c7562ab37eb8d31067cc5f95ec68ee5a565"} Jul 16 18:17:09.165949 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:09.165592 2582 scope.go:117] "RemoveContainer" containerID="83e0c964c16f9dd7a624ff4f6280db9723996517bf1546c9c5fc421bc4bd47b9" Jul 16 18:17:09.205211 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:09.205175 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7"] Jul 16 18:17:09.207548 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:09.207523 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7c2a35/sub-test-model-7c2a35-kserve-59cb89df4f-jr5s7"] Jul 16 18:17:10.653295 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:17:10.653248 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5cd2dc-4977-4423-b094-634218d51fca" path="/var/lib/kubelet/pods/0d5cd2dc-4977-4423-b094-634218d51fca/volumes" Jul 16 18:18:13.340721 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.340682 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-140040552684336"] Jul 16 18:18:13.341140 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.341100 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5cd2dc-4977-4423-b094-634218d51fca" containerName="main" Jul 16 18:18:13.341140 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.341113 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5cd2dc-4977-4423-b094-634218d51fca" containerName="main" Jul 16 18:18:13.341219 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.341209 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5cd2dc-4977-4423-b094-634218d51fca" containerName="main" Jul 16 18:18:13.344353 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.344331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:13.347377 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.347353 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xl26f\"" Jul 16 18:18:13.350095 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.350070 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040552684336"] Jul 16 18:18:13.493353 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.493316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xptk\" (UniqueName: \"kubernetes.io/projected/04185a84-417a-44e2-bfcb-75e8c1b3f662-kube-api-access-2xptk\") pod \"test-curl-7579-140040552684336\" (UID: \"04185a84-417a-44e2-bfcb-75e8c1b3f662\") " pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:13.594523 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.594425 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xptk\" (UniqueName: \"kubernetes.io/projected/04185a84-417a-44e2-bfcb-75e8c1b3f662-kube-api-access-2xptk\") pod \"test-curl-7579-140040552684336\" (UID: \"04185a84-417a-44e2-bfcb-75e8c1b3f662\") " pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:13.604269 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.604241 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xptk\" (UniqueName: \"kubernetes.io/projected/04185a84-417a-44e2-bfcb-75e8c1b3f662-kube-api-access-2xptk\") pod \"test-curl-7579-140040552684336\" (UID: \"04185a84-417a-44e2-bfcb-75e8c1b3f662\") " pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:13.655069 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.655014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:13.809612 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.809583 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040552684336"] Jul 16 18:18:13.811830 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:18:13.811804 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04185a84_417a_44e2_bfcb_75e8c1b3f662.slice/crio-208bbc51dbb37ba36920b05590a17e336fd31ccdb064a603a13315bb430a951b WatchSource:0}: Error finding container 208bbc51dbb37ba36920b05590a17e336fd31ccdb064a603a13315bb430a951b: Status 404 returned error can't find the container with id 208bbc51dbb37ba36920b05590a17e336fd31ccdb064a603a13315bb430a951b Jul 16 18:18:13.813150 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:13.813131 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 18:18:14.542241 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:14.542211 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040552684336" event={"ID":"04185a84-417a-44e2-bfcb-75e8c1b3f662","Type":"ContainerStarted","Data":"208bbc51dbb37ba36920b05590a17e336fd31ccdb064a603a13315bb430a951b"} Jul 16 18:18:15.549767 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:15.549730 2582 generic.go:356] "Generic (PLEG): container finished" podID="04185a84-417a-44e2-bfcb-75e8c1b3f662" containerID="be7d8d36cfd129ac321e106f93b6ac54f95c2b87892cfb730eaaf425760e5fc0" exitCode=0 Jul 16 18:18:15.550208 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:15.549812 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040552684336" event={"ID":"04185a84-417a-44e2-bfcb-75e8c1b3f662","Type":"ContainerDied","Data":"be7d8d36cfd129ac321e106f93b6ac54f95c2b87892cfb730eaaf425760e5fc0"} Jul 16 18:18:16.711139 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.711110 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:16.752167 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.752110 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7579-140040552684336_04185a84-417a-44e2-bfcb-75e8c1b3f662/test-curl-7579-140040552684336/0.log" Jul 16 18:18:16.777548 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.777506 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-140040552684336"] Jul 16 18:18:16.783697 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.783659 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-140040552684336"] Jul 16 18:18:16.825093 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.825004 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xptk\" (UniqueName: \"kubernetes.io/projected/04185a84-417a-44e2-bfcb-75e8c1b3f662-kube-api-access-2xptk\") pod \"04185a84-417a-44e2-bfcb-75e8c1b3f662\" (UID: \"04185a84-417a-44e2-bfcb-75e8c1b3f662\") " Jul 16 18:18:16.827338 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.827306 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04185a84-417a-44e2-bfcb-75e8c1b3f662-kube-api-access-2xptk" (OuterVolumeSpecName: "kube-api-access-2xptk") pod "04185a84-417a-44e2-bfcb-75e8c1b3f662" (UID: "04185a84-417a-44e2-bfcb-75e8c1b3f662"). InnerVolumeSpecName "kube-api-access-2xptk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:18:16.915609 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.915568 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-140040552685232"] Jul 16 18:18:16.916065 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.916045 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04185a84-417a-44e2-bfcb-75e8c1b3f662" containerName="test-curl-7579-140040552684336" Jul 16 18:18:16.916154 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.916067 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04185a84-417a-44e2-bfcb-75e8c1b3f662" containerName="test-curl-7579-140040552684336" Jul 16 18:18:16.916211 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.916190 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04185a84-417a-44e2-bfcb-75e8c1b3f662" containerName="test-curl-7579-140040552684336" Jul 16 18:18:16.919246 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.919223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040552685232" Jul 16 18:18:16.925942 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.925885 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040552685232"] Jul 16 18:18:16.926402 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:16.926374 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xptk\" (UniqueName: \"kubernetes.io/projected/04185a84-417a-44e2-bfcb-75e8c1b3f662-kube-api-access-2xptk\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:18:17.027647 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:17.027605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwtz\" (UniqueName: \"kubernetes.io/projected/161bb351-95e5-4651-a184-61b97dd479eb-kube-api-access-hnwtz\") pod \"test-curl-7579-140040552685232\" (UID: \"161bb351-95e5-4651-a184-61b97dd479eb\") " pod="opendatahub/test-curl-7579-140040552685232" Jul 16 18:18:17.128742 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:17.128643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwtz\" (UniqueName: \"kubernetes.io/projected/161bb351-95e5-4651-a184-61b97dd479eb-kube-api-access-hnwtz\") pod \"test-curl-7579-140040552685232\" (UID: \"161bb351-95e5-4651-a184-61b97dd479eb\") " pod="opendatahub/test-curl-7579-140040552685232" Jul 16 18:18:17.138914 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:17.138887 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwtz\" (UniqueName: \"kubernetes.io/projected/161bb351-95e5-4651-a184-61b97dd479eb-kube-api-access-hnwtz\") pod \"test-curl-7579-140040552685232\" (UID: \"161bb351-95e5-4651-a184-61b97dd479eb\") " pod="opendatahub/test-curl-7579-140040552685232" Jul 16 18:18:17.229959 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:17.229919 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040552685232" Jul 16 18:18:17.377232 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:17.377207 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040552685232"] Jul 16 18:18:17.378854 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:18:17.378788 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod161bb351_95e5_4651_a184_61b97dd479eb.slice/crio-1dc8cb4525e59e301c49feb43b03ce95f98dc9f767d525fb538275eaca63733b WatchSource:0}: Error finding container 1dc8cb4525e59e301c49feb43b03ce95f98dc9f767d525fb538275eaca63733b: Status 404 returned error can't find the container with id 1dc8cb4525e59e301c49feb43b03ce95f98dc9f767d525fb538275eaca63733b Jul 16 18:18:17.564977 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:17.564942 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040552685232" event={"ID":"161bb351-95e5-4651-a184-61b97dd479eb","Type":"ContainerStarted","Data":"1dc8cb4525e59e301c49feb43b03ce95f98dc9f767d525fb538275eaca63733b"} Jul 16 18:18:17.566109 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:17.566088 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208bbc51dbb37ba36920b05590a17e336fd31ccdb064a603a13315bb430a951b" Jul 16 18:18:17.566240 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:17.566116 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:17.583687 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:18:17.583651 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-140040552684336\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="04185a84-417a-44e2-bfcb-75e8c1b3f662" pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:18.573895 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:18.573860 2582 generic.go:356] "Generic (PLEG): container finished" podID="161bb351-95e5-4651-a184-61b97dd479eb" containerID="94496616e9c64a528ce6f1b210fc4383744949f832ee6c18ee165bd7a9719ce2" exitCode=0 Jul 16 18:18:18.574308 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:18.573938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040552685232" event={"ID":"161bb351-95e5-4651-a184-61b97dd479eb","Type":"ContainerDied","Data":"94496616e9c64a528ce6f1b210fc4383744949f832ee6c18ee165bd7a9719ce2"} Jul 16 18:18:18.576431 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:18:18.576407 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-140040552684336\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="04185a84-417a-44e2-bfcb-75e8c1b3f662" pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:18.653027 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:18.652997 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04185a84-417a-44e2-bfcb-75e8c1b3f662" path="/var/lib/kubelet/pods/04185a84-417a-44e2-bfcb-75e8c1b3f662/volumes" Jul 16 18:18:18.655232 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:18:18.655206 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-140040552684336\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="04185a84-417a-44e2-bfcb-75e8c1b3f662" pod="opendatahub/test-curl-7579-140040552684336" Jul 16 18:18:19.726442 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.726416 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040552685232" Jul 16 18:18:19.750366 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.750334 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7579-140040552685232_161bb351-95e5-4651-a184-61b97dd479eb/test-curl-7579-140040552685232/0.log" Jul 16 18:18:19.781481 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.781438 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-140040552685232"] Jul 16 18:18:19.785888 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.785864 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-140040552685232"] Jul 16 18:18:19.856421 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.856336 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnwtz\" (UniqueName: \"kubernetes.io/projected/161bb351-95e5-4651-a184-61b97dd479eb-kube-api-access-hnwtz\") pod \"161bb351-95e5-4651-a184-61b97dd479eb\" (UID: \"161bb351-95e5-4651-a184-61b97dd479eb\") " Jul 16 18:18:19.858566 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.858540 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161bb351-95e5-4651-a184-61b97dd479eb-kube-api-access-hnwtz" (OuterVolumeSpecName: "kube-api-access-hnwtz") pod "161bb351-95e5-4651-a184-61b97dd479eb" (UID: "161bb351-95e5-4651-a184-61b97dd479eb"). InnerVolumeSpecName "kube-api-access-hnwtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:18:19.957014 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.956968 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwtz\" (UniqueName: \"kubernetes.io/projected/161bb351-95e5-4651-a184-61b97dd479eb-kube-api-access-hnwtz\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:18:19.960777 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.960740 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:19.961153 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.961139 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161bb351-95e5-4651-a184-61b97dd479eb" containerName="test-curl-7579-140040552685232" Jul 16 18:18:19.961200 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.961155 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="161bb351-95e5-4651-a184-61b97dd479eb" containerName="test-curl-7579-140040552685232" Jul 16 18:18:19.961235 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.961222 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="161bb351-95e5-4651-a184-61b97dd479eb" containerName="test-curl-7579-140040552685232" Jul 16 18:18:19.964304 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.964270 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:19.971545 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:19.971325 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:20.058452 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.058408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmgmr\" (UniqueName: \"kubernetes.io/projected/92db349f-ba4a-41e8-a81d-2304f6f9dbfe-kube-api-access-jmgmr\") pod \"test-curl-7579-140040550036800\" (UID: \"92db349f-ba4a-41e8-a81d-2304f6f9dbfe\") " pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:20.159809 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.159704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmgmr\" (UniqueName: \"kubernetes.io/projected/92db349f-ba4a-41e8-a81d-2304f6f9dbfe-kube-api-access-jmgmr\") pod \"test-curl-7579-140040550036800\" (UID: \"92db349f-ba4a-41e8-a81d-2304f6f9dbfe\") " pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:20.169967 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.169939 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmgmr\" (UniqueName: \"kubernetes.io/projected/92db349f-ba4a-41e8-a81d-2304f6f9dbfe-kube-api-access-jmgmr\") pod \"test-curl-7579-140040550036800\" (UID: \"92db349f-ba4a-41e8-a81d-2304f6f9dbfe\") " pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:20.274408 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.274366 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:20.429759 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.429727 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:20.430664 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:18:20.430640 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92db349f_ba4a_41e8_a81d_2304f6f9dbfe.slice/crio-045264804adccc16a26f63c18eeb0f4627ce10d345096d7cde16ead7f4df876e WatchSource:0}: Error finding container 045264804adccc16a26f63c18eeb0f4627ce10d345096d7cde16ead7f4df876e: Status 404 returned error can't find the container with id 045264804adccc16a26f63c18eeb0f4627ce10d345096d7cde16ead7f4df876e Jul 16 18:18:20.588158 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.588122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040550036800" event={"ID":"92db349f-ba4a-41e8-a81d-2304f6f9dbfe","Type":"ContainerStarted","Data":"045264804adccc16a26f63c18eeb0f4627ce10d345096d7cde16ead7f4df876e"} Jul 16 18:18:20.589509 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.589486 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040552685232" Jul 16 18:18:20.589640 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.589528 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dc8cb4525e59e301c49feb43b03ce95f98dc9f767d525fb538275eaca63733b" Jul 16 18:18:20.600345 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:18:20.600306 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-140040552685232\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="161bb351-95e5-4651-a184-61b97dd479eb" pod="opendatahub/test-curl-7579-140040552685232" Jul 16 18:18:20.651479 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:20.651445 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161bb351-95e5-4651-a184-61b97dd479eb" path="/var/lib/kubelet/pods/161bb351-95e5-4651-a184-61b97dd479eb/volumes" Jul 16 18:18:21.597783 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:21.597744 2582 generic.go:356] "Generic (PLEG): container finished" podID="92db349f-ba4a-41e8-a81d-2304f6f9dbfe" containerID="13b42f98cdfe4f94983696dbe795d0b01e991554f827f8bba0396bf40e674b08" exitCode=0 Jul 16 18:18:21.598191 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:21.597831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040550036800" event={"ID":"92db349f-ba4a-41e8-a81d-2304f6f9dbfe","Type":"ContainerDied","Data":"13b42f98cdfe4f94983696dbe795d0b01e991554f827f8bba0396bf40e674b08"} Jul 16 18:18:22.759462 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.759430 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:22.784605 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.784078 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7579-140040550036800_92db349f-ba4a-41e8-a81d-2304f6f9dbfe/test-curl-7579-140040550036800/0.log" Jul 16 18:18:22.811491 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.811458 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:22.818752 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.818723 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:22.885492 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.885387 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmgmr\" (UniqueName: \"kubernetes.io/projected/92db349f-ba4a-41e8-a81d-2304f6f9dbfe-kube-api-access-jmgmr\") pod \"92db349f-ba4a-41e8-a81d-2304f6f9dbfe\" (UID: \"92db349f-ba4a-41e8-a81d-2304f6f9dbfe\") " Jul 16 18:18:22.887714 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.887675 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92db349f-ba4a-41e8-a81d-2304f6f9dbfe-kube-api-access-jmgmr" (OuterVolumeSpecName: "kube-api-access-jmgmr") pod "92db349f-ba4a-41e8-a81d-2304f6f9dbfe" (UID: "92db349f-ba4a-41e8-a81d-2304f6f9dbfe"). InnerVolumeSpecName "kube-api-access-jmgmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:18:22.940747 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.940706 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:22.941175 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.941156 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92db349f-ba4a-41e8-a81d-2304f6f9dbfe" containerName="test-curl-7579-140040550036800" Jul 16 18:18:22.941269 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.941179 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="92db349f-ba4a-41e8-a81d-2304f6f9dbfe" containerName="test-curl-7579-140040550036800" Jul 16 18:18:22.941365 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.941326 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="92db349f-ba4a-41e8-a81d-2304f6f9dbfe" containerName="test-curl-7579-140040550036800" Jul 16 18:18:22.944486 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.944465 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:22.949435 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.949407 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:22.986111 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:22.986082 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmgmr\" (UniqueName: \"kubernetes.io/projected/92db349f-ba4a-41e8-a81d-2304f6f9dbfe-kube-api-access-jmgmr\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:18:23.086818 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.086768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmbs\" (UniqueName: \"kubernetes.io/projected/330c7ffc-4dac-4402-b4c9-9f8e315ccb3f-kube-api-access-wtmbs\") pod \"test-curl-7579-140040550036800\" (UID: \"330c7ffc-4dac-4402-b4c9-9f8e315ccb3f\") " pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:23.188460 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.188368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmbs\" (UniqueName: \"kubernetes.io/projected/330c7ffc-4dac-4402-b4c9-9f8e315ccb3f-kube-api-access-wtmbs\") pod \"test-curl-7579-140040550036800\" (UID: \"330c7ffc-4dac-4402-b4c9-9f8e315ccb3f\") " pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:23.199878 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.199842 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmbs\" (UniqueName: \"kubernetes.io/projected/330c7ffc-4dac-4402-b4c9-9f8e315ccb3f-kube-api-access-wtmbs\") pod \"test-curl-7579-140040550036800\" (UID: \"330c7ffc-4dac-4402-b4c9-9f8e315ccb3f\") " pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:23.254934 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.254900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:23.396641 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.396616 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:23.398424 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:18:23.398399 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod330c7ffc_4dac_4402_b4c9_9f8e315ccb3f.slice/crio-c2ab9cdb98dee93dd508be77df5994870c88b509d2a2b2bf5a61bfa25ff69220 WatchSource:0}: Error finding container c2ab9cdb98dee93dd508be77df5994870c88b509d2a2b2bf5a61bfa25ff69220: Status 404 returned error can't find the container with id c2ab9cdb98dee93dd508be77df5994870c88b509d2a2b2bf5a61bfa25ff69220 Jul 16 18:18:23.611124 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.611087 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040550036800" event={"ID":"330c7ffc-4dac-4402-b4c9-9f8e315ccb3f","Type":"ContainerStarted","Data":"c2ab9cdb98dee93dd508be77df5994870c88b509d2a2b2bf5a61bfa25ff69220"} Jul 16 18:18:23.612341 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.612317 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="045264804adccc16a26f63c18eeb0f4627ce10d345096d7cde16ead7f4df876e" Jul 16 18:18:23.612341 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.612328 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:23.624409 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:23.624369 2582 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7579-140040550036800" oldPodUID="92db349f-ba4a-41e8-a81d-2304f6f9dbfe" podUID="330c7ffc-4dac-4402-b4c9-9f8e315ccb3f" Jul 16 18:18:24.619381 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:24.619343 2582 generic.go:356] "Generic (PLEG): container finished" podID="330c7ffc-4dac-4402-b4c9-9f8e315ccb3f" containerID="69d006328f94b62f15ceca2b4c2fb2db4593745bd4e09629097d6f84d2bf7d98" exitCode=0 Jul 16 18:18:24.619752 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:24.619397 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040550036800" event={"ID":"330c7ffc-4dac-4402-b4c9-9f8e315ccb3f","Type":"ContainerDied","Data":"69d006328f94b62f15ceca2b4c2fb2db4593745bd4e09629097d6f84d2bf7d98"} Jul 16 18:18:24.651713 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:24.651676 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92db349f-ba4a-41e8-a81d-2304f6f9dbfe" path="/var/lib/kubelet/pods/92db349f-ba4a-41e8-a81d-2304f6f9dbfe/volumes" Jul 16 18:18:25.767097 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.767073 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:25.794502 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.794460 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7579-140040550036800_330c7ffc-4dac-4402-b4c9-9f8e315ccb3f/test-curl-7579-140040550036800/0.log" Jul 16 18:18:25.823564 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.823532 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:25.826178 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.826150 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-140040550036800"] Jul 16 18:18:25.913375 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.913264 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtmbs\" (UniqueName: \"kubernetes.io/projected/330c7ffc-4dac-4402-b4c9-9f8e315ccb3f-kube-api-access-wtmbs\") pod \"330c7ffc-4dac-4402-b4c9-9f8e315ccb3f\" (UID: \"330c7ffc-4dac-4402-b4c9-9f8e315ccb3f\") " Jul 16 18:18:25.915582 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.915552 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330c7ffc-4dac-4402-b4c9-9f8e315ccb3f-kube-api-access-wtmbs" (OuterVolumeSpecName: "kube-api-access-wtmbs") pod "330c7ffc-4dac-4402-b4c9-9f8e315ccb3f" (UID: "330c7ffc-4dac-4402-b4c9-9f8e315ccb3f"). InnerVolumeSpecName "kube-api-access-wtmbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:18:25.955390 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.955361 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-140040550037808"] Jul 16 18:18:25.955803 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.955788 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330c7ffc-4dac-4402-b4c9-9f8e315ccb3f" containerName="test-curl-7579-140040550036800" Jul 16 18:18:25.955882 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.955805 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="330c7ffc-4dac-4402-b4c9-9f8e315ccb3f" containerName="test-curl-7579-140040550036800" Jul 16 18:18:25.955882 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.955878 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="330c7ffc-4dac-4402-b4c9-9f8e315ccb3f" containerName="test-curl-7579-140040550036800" Jul 16 18:18:25.958763 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.958745 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550037808" Jul 16 18:18:25.963961 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:25.963933 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040550037808"] Jul 16 18:18:26.014493 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.014458 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtmbs\" (UniqueName: \"kubernetes.io/projected/330c7ffc-4dac-4402-b4c9-9f8e315ccb3f-kube-api-access-wtmbs\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:18:26.115120 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.115078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhp7v\" (UniqueName: \"kubernetes.io/projected/c8305e5b-bb61-45a2-a227-06ed50f4d00a-kube-api-access-dhp7v\") pod \"test-curl-7579-140040550037808\" (UID: \"c8305e5b-bb61-45a2-a227-06ed50f4d00a\") " pod="opendatahub/test-curl-7579-140040550037808" Jul 16 18:18:26.216695 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.216600 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhp7v\" (UniqueName: \"kubernetes.io/projected/c8305e5b-bb61-45a2-a227-06ed50f4d00a-kube-api-access-dhp7v\") pod \"test-curl-7579-140040550037808\" (UID: \"c8305e5b-bb61-45a2-a227-06ed50f4d00a\") " pod="opendatahub/test-curl-7579-140040550037808" Jul 16 18:18:26.226765 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.226729 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhp7v\" (UniqueName: \"kubernetes.io/projected/c8305e5b-bb61-45a2-a227-06ed50f4d00a-kube-api-access-dhp7v\") pod \"test-curl-7579-140040550037808\" (UID: \"c8305e5b-bb61-45a2-a227-06ed50f4d00a\") " pod="opendatahub/test-curl-7579-140040550037808" Jul 16 18:18:26.268794 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.268755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550037808" Jul 16 18:18:26.416324 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.415618 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-140040550037808"] Jul 16 18:18:26.631940 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.631904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040550037808" event={"ID":"c8305e5b-bb61-45a2-a227-06ed50f4d00a","Type":"ContainerStarted","Data":"5bed5693a48b95af6a3d2539cf985cde37831f55fc8211913fe8e870d738e0b7"} Jul 16 18:18:26.633247 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.633227 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ab9cdb98dee93dd508be77df5994870c88b509d2a2b2bf5a61bfa25ff69220" Jul 16 18:18:26.633247 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.633251 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:26.644230 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:18:26.644200 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-140040550036800\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="330c7ffc-4dac-4402-b4c9-9f8e315ccb3f" pod="opendatahub/test-curl-7579-140040550036800" Jul 16 18:18:26.651716 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:26.651689 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="330c7ffc-4dac-4402-b4c9-9f8e315ccb3f" path="/var/lib/kubelet/pods/330c7ffc-4dac-4402-b4c9-9f8e315ccb3f/volumes" Jul 16 18:18:27.639906 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:27.639859 2582 generic.go:356] "Generic (PLEG): container finished" podID="c8305e5b-bb61-45a2-a227-06ed50f4d00a" containerID="60a8341771c32ce4e8a265c413125ccc803f7a6b7b1936b4afe871d889c10436" exitCode=0 Jul 16 18:18:27.640366 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:27.639941 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-140040550037808" event={"ID":"c8305e5b-bb61-45a2-a227-06ed50f4d00a","Type":"ContainerDied","Data":"60a8341771c32ce4e8a265c413125ccc803f7a6b7b1936b4afe871d889c10436"} Jul 16 18:18:28.797330 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:28.797300 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550037808" Jul 16 18:18:28.828035 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:28.828010 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7579-140040550037808_c8305e5b-bb61-45a2-a227-06ed50f4d00a/test-curl-7579-140040550037808/0.log" Jul 16 18:18:28.858883 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:28.858846 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-140040550037808"] Jul 16 18:18:28.863653 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:28.863626 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-140040550037808"] Jul 16 18:18:28.942531 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:28.942440 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhp7v\" (UniqueName: \"kubernetes.io/projected/c8305e5b-bb61-45a2-a227-06ed50f4d00a-kube-api-access-dhp7v\") pod \"c8305e5b-bb61-45a2-a227-06ed50f4d00a\" (UID: \"c8305e5b-bb61-45a2-a227-06ed50f4d00a\") " Jul 16 18:18:28.944690 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:28.944656 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8305e5b-bb61-45a2-a227-06ed50f4d00a-kube-api-access-dhp7v" (OuterVolumeSpecName: "kube-api-access-dhp7v") pod "c8305e5b-bb61-45a2-a227-06ed50f4d00a" (UID: "c8305e5b-bb61-45a2-a227-06ed50f4d00a"). InnerVolumeSpecName "kube-api-access-dhp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:18:29.043394 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:29.043359 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhp7v\" (UniqueName: \"kubernetes.io/projected/c8305e5b-bb61-45a2-a227-06ed50f4d00a-kube-api-access-dhp7v\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:18:29.652048 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:29.652018 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-140040550037808" Jul 16 18:18:29.652048 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:29.652032 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bed5693a48b95af6a3d2539cf985cde37831f55fc8211913fe8e870d738e0b7" Jul 16 18:18:29.662516 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:18:29.662477 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-140040550037808\" is forbidden: User \"system:node:ip-10-0-136-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-229.ec2.internal' and this object" podUID="c8305e5b-bb61-45a2-a227-06ed50f4d00a" pod="opendatahub/test-curl-7579-140040550037808" Jul 16 18:18:30.654008 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:30.653971 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8305e5b-bb61-45a2-a227-06ed50f4d00a" path="/var/lib/kubelet/pods/c8305e5b-bb61-45a2-a227-06ed50f4d00a/volumes" Jul 16 18:18:42.371961 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.371927 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx"] Jul 16 18:18:42.372475 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.372151 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" podUID="f4b683c8-cc87-4315-97cb-11785b57b22b" containerName="maas-api" containerID="cri-o://be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f" gracePeriod=30 Jul 16 18:18:42.632128 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.632064 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:18:42.662781 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.662752 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcrbp\" (UniqueName: \"kubernetes.io/projected/f4b683c8-cc87-4315-97cb-11785b57b22b-kube-api-access-tcrbp\") pod \"f4b683c8-cc87-4315-97cb-11785b57b22b\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " Jul 16 18:18:42.662781 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.662787 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b683c8-cc87-4315-97cb-11785b57b22b-openshift-service-ca\") pod \"f4b683c8-cc87-4315-97cb-11785b57b22b\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " Jul 16 18:18:42.663015 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.662842 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls\") pod \"f4b683c8-cc87-4315-97cb-11785b57b22b\" (UID: \"f4b683c8-cc87-4315-97cb-11785b57b22b\") " Jul 16 18:18:42.663227 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.663184 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b683c8-cc87-4315-97cb-11785b57b22b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f4b683c8-cc87-4315-97cb-11785b57b22b" (UID: "f4b683c8-cc87-4315-97cb-11785b57b22b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:18:42.665209 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.665176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f4b683c8-cc87-4315-97cb-11785b57b22b" (UID: "f4b683c8-cc87-4315-97cb-11785b57b22b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:18:42.665359 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.665247 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b683c8-cc87-4315-97cb-11785b57b22b-kube-api-access-tcrbp" (OuterVolumeSpecName: "kube-api-access-tcrbp") pod "f4b683c8-cc87-4315-97cb-11785b57b22b" (UID: "f4b683c8-cc87-4315-97cb-11785b57b22b"). InnerVolumeSpecName "kube-api-access-tcrbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:18:42.725999 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.725969 2582 generic.go:356] "Generic (PLEG): container finished" podID="f4b683c8-cc87-4315-97cb-11785b57b22b" containerID="be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f" exitCode=0 Jul 16 18:18:42.726165 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.726018 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" Jul 16 18:18:42.726165 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.726046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" event={"ID":"f4b683c8-cc87-4315-97cb-11785b57b22b","Type":"ContainerDied","Data":"be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f"} Jul 16 18:18:42.726165 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.726100 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx" event={"ID":"f4b683c8-cc87-4315-97cb-11785b57b22b","Type":"ContainerDied","Data":"b83103d74ba9c0a2697ba69dd12d8cd293ab9c1009eaa9040203215ab601b661"} Jul 16 18:18:42.726165 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.726121 2582 scope.go:117] "RemoveContainer" containerID="be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f" Jul 16 18:18:42.735946 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.735929 2582 scope.go:117] "RemoveContainer" containerID="be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f" Jul 16 18:18:42.736240 ip-10-0-136-229 kubenswrapper[2582]: E0716 18:18:42.736219 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f\": container with ID starting with be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f not found: ID does not exist" containerID="be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f" Jul 16 18:18:42.736340 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.736252 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f"} err="failed to get container status \"be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f\": rpc error: code = NotFound desc = could not find container \"be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f\": container with ID starting with be2b5fafca3b515ba6eaab37304a388cdb461c51baca451dfe771b43c4de8f6f not found: ID does not exist" Jul 16 18:18:42.750652 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.750622 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx"] Jul 16 18:18:42.755610 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.755585 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7c2a35-76747b76b8-jbgzx"] Jul 16 18:18:42.763636 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.763615 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b683c8-cc87-4315-97cb-11785b57b22b-maas-api-tls\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:18:42.763636 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.763636 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcrbp\" (UniqueName: \"kubernetes.io/projected/f4b683c8-cc87-4315-97cb-11785b57b22b-kube-api-access-tcrbp\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:18:42.763767 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:42.763649 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b683c8-cc87-4315-97cb-11785b57b22b-openshift-service-ca\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:18:44.651436 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:18:44.651400 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4b683c8-cc87-4315-97cb-11785b57b22b" path="/var/lib/kubelet/pods/f4b683c8-cc87-4315-97cb-11785b57b22b/volumes" Jul 16 18:19:05.645889 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.645850 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4"] Jul 16 18:19:05.646518 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.646088 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" podUID="c92060f3-1e7e-4082-bf85-52484f6e1586" containerName="maas-api" containerID="cri-o://fb398568b6a3b46a592e36b40fa0b7a2a50cb9847ca36f26d3a5f74254f9a908" gracePeriod=30 Jul 16 18:19:05.858569 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.858531 2582 generic.go:356] "Generic (PLEG): container finished" podID="c92060f3-1e7e-4082-bf85-52484f6e1586" containerID="fb398568b6a3b46a592e36b40fa0b7a2a50cb9847ca36f26d3a5f74254f9a908" exitCode=0 Jul 16 18:19:05.858774 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.858611 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" event={"ID":"c92060f3-1e7e-4082-bf85-52484f6e1586","Type":"ContainerDied","Data":"fb398568b6a3b46a592e36b40fa0b7a2a50cb9847ca36f26d3a5f74254f9a908"} Jul 16 18:19:05.907928 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.907854 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:19:05.981971 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.981935 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8twsb\" (UniqueName: \"kubernetes.io/projected/c92060f3-1e7e-4082-bf85-52484f6e1586-kube-api-access-8twsb\") pod \"c92060f3-1e7e-4082-bf85-52484f6e1586\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " Jul 16 18:19:05.982189 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.982053 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls\") pod \"c92060f3-1e7e-4082-bf85-52484f6e1586\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " Jul 16 18:19:05.982189 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.982127 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92060f3-1e7e-4082-bf85-52484f6e1586-openshift-service-ca\") pod \"c92060f3-1e7e-4082-bf85-52484f6e1586\" (UID: \"c92060f3-1e7e-4082-bf85-52484f6e1586\") " Jul 16 18:19:05.982608 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.982576 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92060f3-1e7e-4082-bf85-52484f6e1586-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c92060f3-1e7e-4082-bf85-52484f6e1586" (UID: "c92060f3-1e7e-4082-bf85-52484f6e1586"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:19:05.984465 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.984435 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92060f3-1e7e-4082-bf85-52484f6e1586-kube-api-access-8twsb" (OuterVolumeSpecName: "kube-api-access-8twsb") pod "c92060f3-1e7e-4082-bf85-52484f6e1586" (UID: "c92060f3-1e7e-4082-bf85-52484f6e1586"). InnerVolumeSpecName "kube-api-access-8twsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:19:05.984559 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:05.984526 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c92060f3-1e7e-4082-bf85-52484f6e1586" (UID: "c92060f3-1e7e-4082-bf85-52484f6e1586"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:19:06.083816 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:06.083778 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c92060f3-1e7e-4082-bf85-52484f6e1586-maas-api-tls\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:19:06.083816 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:06.083819 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92060f3-1e7e-4082-bf85-52484f6e1586-openshift-service-ca\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:19:06.084024 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:06.083834 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twsb\" (UniqueName: \"kubernetes.io/projected/c92060f3-1e7e-4082-bf85-52484f6e1586-kube-api-access-8twsb\") on node \"ip-10-0-136-229.ec2.internal\" DevicePath \"\"" Jul 16 18:19:06.865783 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:06.865755 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" Jul 16 18:19:06.866245 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:06.865758 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4" event={"ID":"c92060f3-1e7e-4082-bf85-52484f6e1586","Type":"ContainerDied","Data":"407a500b251c673ef32b05ef6daaa7a5919835ef8a7cc0ed43c47fde86d42d40"} Jul 16 18:19:06.866245 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:06.865899 2582 scope.go:117] "RemoveContainer" containerID="fb398568b6a3b46a592e36b40fa0b7a2a50cb9847ca36f26d3a5f74254f9a908" Jul 16 18:19:06.887756 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:06.887730 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4"] Jul 16 18:19:06.894013 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:06.893989 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d3e547-5c5bd5b79d-dczj4"] Jul 16 18:19:08.652912 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:08.652878 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c92060f3-1e7e-4082-bf85-52484f6e1586" path="/var/lib/kubelet/pods/c92060f3-1e7e-4082-bf85-52484f6e1586/volumes" Jul 16 18:19:19.288344 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:19.288304 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84655594dc-wfpfq_7cbc6f15-9138-4ec3-8091-9ffbd626b6d0/authorino/0.log" Jul 16 18:19:24.999600 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:24.999567 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6wn7f_43fae792-e9bd-4de3-bd7c-8a327a7e40de/manager/0.log" Jul 16 18:19:25.137641 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:25.137610 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pp4dd_7861aee2-fad9-4cc1-a518-268115eb3265/manager/0.log" Jul 16 18:19:25.404622 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:25.404590 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r59fz_7ecc13a7-925c-4045-9da0-51909d1cf391/server/0.log" Jul 16 18:19:25.535490 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:25.535462 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-t6d4r_e9f77167-bd56-464f-b6e8-c41e91b71181/manager/0.log" Jul 16 18:19:25.795859 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:25.795829 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-586f4b4fcb-cd69m_07dce6dc-1128-4309-96c4-984089699d19/copy-manifests/0.log" Jul 16 18:19:25.811684 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:25.811654 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-586f4b4fcb-cd69m_07dce6dc-1128-4309-96c4-984089699d19/manager/0.log" Jul 16 18:19:26.358088 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:26.358057 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d67b469d5-pwdk4_96b6891f-ae9d-416e-bcbe-9f08edfee3cb/maas-api/0.log" Jul 16 18:19:29.048115 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.048074 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd_77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46/extract/0.log" Jul 16 18:19:29.055703 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.055676 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd_77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46/util/0.log" Jul 16 18:19:29.064132 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.064099 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd_77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46/pull/0.log" Jul 16 18:19:29.187827 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.187797 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs_029c6c08-d801-442e-a164-b007387c92bd/util/0.log" Jul 16 18:19:29.196413 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.196385 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs_029c6c08-d801-442e-a164-b007387c92bd/pull/0.log" Jul 16 18:19:29.204416 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.204390 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs_029c6c08-d801-442e-a164-b007387c92bd/extract/0.log" Jul 16 18:19:29.329236 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.329148 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6_ae59f884-9a62-4b89-8570-fe266763d135/util/0.log" Jul 16 18:19:29.340156 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.340128 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6_ae59f884-9a62-4b89-8570-fe266763d135/pull/0.log" Jul 16 18:19:29.349519 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.349493 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6_ae59f884-9a62-4b89-8570-fe266763d135/extract/0.log" Jul 16 18:19:29.474983 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.474948 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc_ac4202e8-0b36-4229-8fa9-ab8784ff48df/extract/0.log" Jul 16 18:19:29.483595 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.483574 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc_ac4202e8-0b36-4229-8fa9-ab8784ff48df/util/0.log" Jul 16 18:19:29.492023 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.491995 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc_ac4202e8-0b36-4229-8fa9-ab8784ff48df/pull/0.log" Jul 16 18:19:29.620435 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.620339 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84655594dc-wfpfq_7cbc6f15-9138-4ec3-8091-9ffbd626b6d0/authorino/0.log" Jul 16 18:19:29.758511 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.758479 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-s87j6_7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97/manager/0.log" Jul 16 18:19:29.884330 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:29.884231 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nqwqb_a478a22d-09f4-4a8e-8bf7-b18673b760e6/manager/0.log" Jul 16 18:19:30.136488 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:30.136396 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xpm4s_751610fa-ccc0-4a31-8110-e5660bb7f146/registry-server/0.log" Jul 16 18:19:30.263998 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:30.263964 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qzb28_9d272945-fa69-48ca-88ef-43d0439890b4/manager/0.log" Jul 16 18:19:30.532622 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:30.532587 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jszsl_9ff6b759-d3d3-4d1e-b216-02a81309f907/manager/0.log" Jul 16 18:19:30.896759 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:30.896655 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46c8gd_cc90a598-4135-43a9-a06e-db32379f92a4/istio-proxy/0.log" Jul 16 18:19:31.143685 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:31.143645 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8cccc594-hpkrl_45b2bca5-c938-45fd-b8e3-b010d365d933/kube-auth-proxy/0.log" Jul 16 18:19:31.390473 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:31.390441 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwrxt_cedcd98b-0520-456c-a82b-e1c9948e8b76/istio-proxy/1.log" Jul 16 18:19:31.637316 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:31.637262 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-8x4ks_003f7852-ea9b-4158-926d-303d18d7d8fc/payload-processing/0.log" Jul 16 18:19:31.765359 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:31.765252 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7798b5fb-gw7rk_e4264197-b690-4ca8-a31f-c6c32cf8eba8/router/0.log" Jul 16 18:19:32.464441 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:32.464402 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbf7x54_45978a28-d484-4a91-aa43-0b602e5683bd/main/0.log" Jul 16 18:19:32.707342 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:32.707288 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-tz9fp_a71f03ac-8e7d-44e5-904d-80b9785eeda3/main/0.log" Jul 16 18:19:40.858324 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:40.858253 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f2hc6_8dd60e07-b295-4af0-b21b-28022c903e65/global-pull-secret-syncer/0.log" Jul 16 18:19:41.009846 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:41.009803 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nxml2_c6703413-e169-4ae5-a4a6-c6b6d93df642/konnectivity-agent/0.log" Jul 16 18:19:41.127631 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:41.127543 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-229.ec2.internal_0b041e55b23cc7cb3930d7a715f578cc/haproxy/0.log" Jul 16 18:19:45.185534 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.185442 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd_77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46/extract/0.log" Jul 16 18:19:45.212814 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.212786 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd_77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46/util/0.log" Jul 16 18:19:45.241596 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.241568 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ltjmd_77616f97-d5ca-4cdc-ba3b-9aa0fe7bae46/pull/0.log" Jul 16 18:19:45.276994 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.276964 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs_029c6c08-d801-442e-a164-b007387c92bd/extract/0.log" Jul 16 18:19:45.309092 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.309061 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs_029c6c08-d801-442e-a164-b007387c92bd/util/0.log" Jul 16 18:19:45.343306 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.343251 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gc8xs_029c6c08-d801-442e-a164-b007387c92bd/pull/0.log" Jul 16 18:19:45.388430 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.388400 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6_ae59f884-9a62-4b89-8570-fe266763d135/extract/0.log" Jul 16 18:19:45.423018 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.422993 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6_ae59f884-9a62-4b89-8570-fe266763d135/util/0.log" Jul 16 18:19:45.461919 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.461839 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dpvz6_ae59f884-9a62-4b89-8570-fe266763d135/pull/0.log" Jul 16 18:19:45.516527 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.516488 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc_ac4202e8-0b36-4229-8fa9-ab8784ff48df/extract/0.log" Jul 16 18:19:45.543714 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.543681 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc_ac4202e8-0b36-4229-8fa9-ab8784ff48df/util/0.log" Jul 16 18:19:45.578484 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:45.578452 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp8fc_ac4202e8-0b36-4229-8fa9-ab8784ff48df/pull/0.log" Jul 16 18:19:46.309236 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:46.309204 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84655594dc-wfpfq_7cbc6f15-9138-4ec3-8091-9ffbd626b6d0/authorino/0.log" Jul 16 18:19:46.389194 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:46.389156 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-s87j6_7b7c2a80-7c0b-48e2-b52d-ac1e0facdc97/manager/0.log" Jul 16 18:19:46.447609 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:46.447583 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nqwqb_a478a22d-09f4-4a8e-8bf7-b18673b760e6/manager/0.log" Jul 16 18:19:46.581686 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:46.581599 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xpm4s_751610fa-ccc0-4a31-8110-e5660bb7f146/registry-server/0.log" Jul 16 18:19:46.851529 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:46.851453 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qzb28_9d272945-fa69-48ca-88ef-43d0439890b4/manager/0.log" Jul 16 18:19:47.007093 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:47.007059 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jszsl_9ff6b759-d3d3-4d1e-b216-02a81309f907/manager/0.log" Jul 16 18:19:48.779575 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:48.779540 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-zvnlg_8debfe7c-6fd2-4ed1-a47b-a916cd09d836/cluster-monitoring-operator/0.log" Jul 16 18:19:49.129846 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:49.129762 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tdfgm_35316ded-1df4-4993-8257-cb72b0ed749f/node-exporter/0.log" Jul 16 18:19:49.156412 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:49.156379 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tdfgm_35316ded-1df4-4993-8257-cb72b0ed749f/kube-rbac-proxy/0.log" Jul 16 18:19:49.189017 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:49.188985 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tdfgm_35316ded-1df4-4993-8257-cb72b0ed749f/init-textfile/0.log" Jul 16 18:19:50.266112 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266076 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5"] Jul 16 18:19:50.266533 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266506 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8305e5b-bb61-45a2-a227-06ed50f4d00a" containerName="test-curl-7579-140040550037808" Jul 16 18:19:50.266533 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266518 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8305e5b-bb61-45a2-a227-06ed50f4d00a" containerName="test-curl-7579-140040550037808" Jul 16 18:19:50.266533 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266534 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c92060f3-1e7e-4082-bf85-52484f6e1586" containerName="maas-api" Jul 16 18:19:50.266637 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266541 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c92060f3-1e7e-4082-bf85-52484f6e1586" containerName="maas-api" Jul 16 18:19:50.266637 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266550 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b683c8-cc87-4315-97cb-11785b57b22b" containerName="maas-api" Jul 16 18:19:50.266637 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266555 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b683c8-cc87-4315-97cb-11785b57b22b" containerName="maas-api" Jul 16 18:19:50.266637 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266622 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c92060f3-1e7e-4082-bf85-52484f6e1586" containerName="maas-api" Jul 16 18:19:50.266637 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266631 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b683c8-cc87-4315-97cb-11785b57b22b" containerName="maas-api" Jul 16 18:19:50.266637 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.266637 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8305e5b-bb61-45a2-a227-06ed50f4d00a" containerName="test-curl-7579-140040550037808" Jul 16 18:19:50.269901 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.269877 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.272655 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.272632 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w8h8n\"/\"default-dockercfg-474ns\"" Jul 16 18:19:50.272794 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.272632 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w8h8n\"/\"openshift-service-ca.crt\"" Jul 16 18:19:50.274217 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.274195 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w8h8n\"/\"kube-root-ca.crt\"" Jul 16 18:19:50.278165 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.278138 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5"] Jul 16 18:19:50.379549 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.379513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-proc\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.379549 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.379553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-lib-modules\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.379777 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.379578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5x8\" (UniqueName: \"kubernetes.io/projected/5ae67baa-407a-428d-88dd-8fbf8c75f461-kube-api-access-sb5x8\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.379777 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.379612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-podres\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.379777 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.379635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-sys\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480135 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-proc\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480360 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-lib-modules\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480360 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480163 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5x8\" (UniqueName: \"kubernetes.io/projected/5ae67baa-407a-428d-88dd-8fbf8c75f461-kube-api-access-sb5x8\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480360 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480190 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-proc\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480360 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-podres\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480360 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480306 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-sys\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480360 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480341 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-lib-modules\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480360 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-podres\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.480691 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.480379 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ae67baa-407a-428d-88dd-8fbf8c75f461-sys\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.490317 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.490275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5x8\" (UniqueName: \"kubernetes.io/projected/5ae67baa-407a-428d-88dd-8fbf8c75f461-kube-api-access-sb5x8\") pod \"perf-node-gather-daemonset-c7mw5\" (UID: \"5ae67baa-407a-428d-88dd-8fbf8c75f461\") " pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.580895 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.580859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:50.732620 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:50.732587 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5"] Jul 16 18:19:50.733794 ip-10-0-136-229 kubenswrapper[2582]: W0716 18:19:50.733766 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ae67baa_407a_428d_88dd_8fbf8c75f461.slice/crio-5b1b5ba9cb97d41d44e5a1608dbcb9dedeaa458bc30c81bba8f217cde65ac2c4 WatchSource:0}: Error finding container 5b1b5ba9cb97d41d44e5a1608dbcb9dedeaa458bc30c81bba8f217cde65ac2c4: Status 404 returned error can't find the container with id 5b1b5ba9cb97d41d44e5a1608dbcb9dedeaa458bc30c81bba8f217cde65ac2c4 Jul 16 18:19:51.118779 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:51.118732 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" event={"ID":"5ae67baa-407a-428d-88dd-8fbf8c75f461","Type":"ContainerStarted","Data":"d0bc3c0f6c0f3e3c9550b84174538e245791335d459a585620b072175709243d"} Jul 16 18:19:51.118965 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:51.118804 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:51.118965 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:51.118826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" event={"ID":"5ae67baa-407a-428d-88dd-8fbf8c75f461","Type":"ContainerStarted","Data":"5b1b5ba9cb97d41d44e5a1608dbcb9dedeaa458bc30c81bba8f217cde65ac2c4"} Jul 16 18:19:51.139125 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:51.139074 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" podStartSLOduration=1.139056952 podStartE2EDuration="1.139056952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:19:51.137398707 +0000 UTC m=+3052.889038266" watchObservedRunningTime="2026-07-16 18:19:51.139056952 +0000 UTC m=+3052.890696510" Jul 16 18:19:52.134017 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:52.133982 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-sxqc5_47d72ba2-137d-4c24-a9b2-62f8e2db62fc/download-server/0.log" Jul 16 18:19:52.745877 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:52.745835 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-gfhb5_38a67d8c-f26e-4d5e-b772-2ea4db4b6904/volume-data-source-validator/0.log" Jul 16 18:19:53.940384 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:53.940342 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztxrd_ca0ed375-c645-4cc1-9264-67043142deb5/dns/0.log" Jul 16 18:19:53.976709 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:53.976678 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztxrd_ca0ed375-c645-4cc1-9264-67043142deb5/kube-rbac-proxy/0.log" Jul 16 18:19:54.020393 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:54.020362 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c2rnv_42b15b11-e94a-4155-b9b0-1e5f5cf58c92/dns-node-resolver/0.log" Jul 16 18:19:54.670870 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:54.670831 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86896dd949-lcm9g_2dd999b4-ca4a-4082-b79b-f31a5700b625/registry/0.log" Jul 16 18:19:54.697656 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:54.697624 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-88s2w_636d2c81-02e6-4235-a858-d3993e5dba2c/node-ca/0.log" Jul 16 18:19:55.778145 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:55.778117 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46c8gd_cc90a598-4135-43a9-a06e-db32379f92a4/istio-proxy/0.log" Jul 16 18:19:55.957054 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:55.957025 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8cccc594-hpkrl_45b2bca5-c938-45fd-b8e3-b010d365d933/kube-auth-proxy/0.log" Jul 16 18:19:56.024119 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:56.024089 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwrxt_cedcd98b-0520-456c-a82b-e1c9948e8b76/istio-proxy/0.log" Jul 16 18:19:56.081782 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:56.081688 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwrxt_cedcd98b-0520-456c-a82b-e1c9948e8b76/istio-proxy/1.log" Jul 16 18:19:56.174922 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:56.174889 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-8x4ks_003f7852-ea9b-4158-926d-303d18d7d8fc/payload-processing/0.log" Jul 16 18:19:56.202245 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:56.202210 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7798b5fb-gw7rk_e4264197-b690-4ca8-a31f-c6c32cf8eba8/router/0.log" Jul 16 18:19:56.899217 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:56.899193 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6k2d_8cac65f2-474b-4aa7-80d9-a7530cebfbf5/serve-healthcheck-canary/0.log" Jul 16 18:19:57.136074 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:57.136048 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w8h8n/perf-node-gather-daemonset-c7mw5" Jul 16 18:19:57.682388 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:57.682360 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qpd2g_4764c8fb-7014-456c-9ebb-f22b750cb21e/kube-rbac-proxy/0.log" Jul 16 18:19:57.707438 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:57.707406 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qpd2g_4764c8fb-7014-456c-9ebb-f22b750cb21e/exporter/0.log" Jul 16 18:19:57.735954 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:57.735921 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qpd2g_4764c8fb-7014-456c-9ebb-f22b750cb21e/extractor/0.log" Jul 16 18:19:59.996652 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:19:59.996625 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6wn7f_43fae792-e9bd-4de3-bd7c-8a327a7e40de/manager/0.log" Jul 16 18:20:00.331188 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.331150 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pp4dd_7861aee2-fad9-4cc1-a518-268115eb3265/manager/0.log" Jul 16 18:20:00.483455 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.483373 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r59fz_7ecc13a7-925c-4045-9da0-51909d1cf391/server/0.log" Jul 16 18:20:00.545851 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.545815 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46c8gd_cc90a598-4135-43a9-a06e-db32379f92a4/istio-proxy/0.log" Jul 16 18:20:00.600555 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.600517 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-t6d4r_e9f77167-bd56-464f-b6e8-c41e91b71181/manager/0.log" Jul 16 18:20:00.701770 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.701736 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-586f4b4fcb-cd69m_07dce6dc-1128-4309-96c4-984089699d19/manager/0.log" Jul 16 18:20:00.726422 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.726373 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8cccc594-hpkrl_45b2bca5-c938-45fd-b8e3-b010d365d933/kube-auth-proxy/0.log" Jul 16 18:20:00.733380 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.733354 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-586f4b4fcb-cd69m_07dce6dc-1128-4309-96c4-984089699d19/copy-manifests/0.log" Jul 16 18:20:00.780625 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.780591 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwrxt_cedcd98b-0520-456c-a82b-e1c9948e8b76/istio-proxy/0.log" Jul 16 18:20:00.840265 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.840228 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwrxt_cedcd98b-0520-456c-a82b-e1c9948e8b76/istio-proxy/1.log" Jul 16 18:20:00.904329 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.904304 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-8x4ks_003f7852-ea9b-4158-926d-303d18d7d8fc/payload-processing/0.log" Jul 16 18:20:00.930311 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:00.929208 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7798b5fb-gw7rk_e4264197-b690-4ca8-a31f-c6c32cf8eba8/router/0.log" Jul 16 18:20:02.276892 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:02.276857 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46c8gd_cc90a598-4135-43a9-a06e-db32379f92a4/istio-proxy/0.log" Jul 16 18:20:02.451100 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:02.451069 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8cccc594-hpkrl_45b2bca5-c938-45fd-b8e3-b010d365d933/kube-auth-proxy/0.log" Jul 16 18:20:02.497656 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:02.497624 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwrxt_cedcd98b-0520-456c-a82b-e1c9948e8b76/istio-proxy/0.log" Jul 16 18:20:02.556345 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:02.556314 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwrxt_cedcd98b-0520-456c-a82b-e1c9948e8b76/istio-proxy/1.log" Jul 16 18:20:02.619477 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:02.619447 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-8x4ks_003f7852-ea9b-4158-926d-303d18d7d8fc/payload-processing/0.log" Jul 16 18:20:02.641987 ip-10-0-136-229 kubenswrapper[2582]: I0716 18:20:02.641947 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7798b5fb-gw7rk_e4264197-b690-4ca8-a31f-c6c32cf8eba8/router/0.log"