Jul 14 18:08:13.916409 ip-10-0-129-146 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 18:08:13.916420 ip-10-0-129-146 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 18:08:13.916429 ip-10-0-129-146 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 18:08:13.916738 ip-10-0-129-146 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 18:08:24.096030 ip-10-0-129-146 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 18:08:24.096048 ip-10-0-129-146 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 10bf8a7a7aff4326bc024cdd29b091da -- Jul 14 18:10:00.440841 ip-10-0-129-146 systemd[1]: Starting Kubernetes Kubelet... Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.897899 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902009 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902023 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902027 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902031 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902034 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902037 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902040 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:00.907639 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902043 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902046 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902049 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902052 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902055 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902058 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902060 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902063 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902065 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902068 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902084 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902086 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902089 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902092 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902094 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902097 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902100 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902102 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902105 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902109 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:00.925010 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902111 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902114 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902117 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902119 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902122 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902124 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902127 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902130 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902132 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902135 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902137 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902139 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902142 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902144 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902147 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902149 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902152 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902155 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902158 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902160 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:00.926962 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902163 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902165 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902168 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902176 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902179 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902182 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902185 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902187 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902190 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902193 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902195 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902198 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902201 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902203 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902206 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902208 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902211 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902213 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902216 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902218 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:00.951406 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902220 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902223 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902225 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902228 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902230 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902233 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902236 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902238 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902241 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902244 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902247 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902249 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902253 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902255 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902258 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902260 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902269 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902271 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902274 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:00.953499 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902278 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902281 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902284 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902286 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902289 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902292 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902294 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902298 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902301 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902304 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902307 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902310 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.902313 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906276 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906287 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906291 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906295 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906298 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906300 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906303 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:00.957436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906305 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906309 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906311 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906314 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906316 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906319 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906322 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906324 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906327 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906330 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906332 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906335 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906338 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906340 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906343 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906345 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906348 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906351 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906354 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:00.974343 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906357 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906361 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906365 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906368 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906371 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906375 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906377 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906380 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906382 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906385 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906387 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906390 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906393 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906395 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906398 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906400 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906403 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906405 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906408 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906410 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:00.976959 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906413 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906428 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906431 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906433 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906445 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906447 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906450 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906453 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906455 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906458 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906460 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906463 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906465 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906468 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906471 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906473 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906475 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906478 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906480 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:01.135395 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906483 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:00.987311 ip-10-0-129-146 systemd[1]: Started Kubernetes Kubelet. Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906488 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906492 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906495 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906498 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906500 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906503 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906505 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906507 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906510 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906512 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906515 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906517 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906520 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906524 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906527 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906529 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906532 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906541 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906544 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:01.160592 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906547 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906573 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906577 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906585 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906588 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906591 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906594 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906598 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906600 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906603 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906606 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906608 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906611 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.906613 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907971 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907979 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907982 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907985 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907988 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:01.163058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907991 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907994 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907997 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.907999 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908002 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908005 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908007 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908012 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908014 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908017 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908019 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908027 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908030 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908033 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908035 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908038 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908040 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908044 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908047 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908051 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:01.166097 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908054 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908057 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908059 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908062 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908064 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908067 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908070 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908072 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908075 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908077 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908080 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908082 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908084 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908087 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908089 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908092 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908094 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908096 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908099 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908103 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:01.166916 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908105 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908108 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908110 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908113 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908116 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908119 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908121 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908124 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908126 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908129 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908131 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908134 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908137 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908140 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908142 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908145 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908148 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908150 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908152 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908155 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:01.169120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908158 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908160 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908163 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908165 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908167 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908170 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908172 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908175 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908177 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908179 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908182 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908186 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908188 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908191 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908194 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908198 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908201 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908204 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908207 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:01.169776 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908210 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908214 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908217 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908220 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908223 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908226 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908228 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908231 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908233 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908236 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908238 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908241 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908243 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908245 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908248 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908320 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908328 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908334 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908339 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908343 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 18:10:01.369342 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908346 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908350 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908354 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908357 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908362 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908365 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908368 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908372 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908374 2579 flags.go:64] FLAG: --cgroup-root="" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908377 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908380 2579 flags.go:64] FLAG: --client-ca-file="" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908384 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908387 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908391 2579 flags.go:64] FLAG: --cluster-domain="" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908394 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908397 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908400 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908403 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908407 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908410 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908413 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908416 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908419 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908422 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 18:10:01.369997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908425 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908428 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908430 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908434 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908437 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908440 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908443 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908446 2579 flags.go:64] FLAG: --enable-server="true" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908449 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908454 2579 flags.go:64] FLAG: --event-burst="100" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908457 2579 flags.go:64] FLAG: --event-qps="50" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908460 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908464 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908467 2579 flags.go:64] FLAG: --eviction-hard="" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908475 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908478 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908481 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908484 2579 flags.go:64] FLAG: --eviction-soft="" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908486 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908489 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908492 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908495 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908498 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908501 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908506 2579 flags.go:64] FLAG: --feature-gates="" Jul 14 18:10:01.370699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908510 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908513 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908516 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908519 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908522 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908525 2579 flags.go:64] FLAG: --help="false" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908528 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908531 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908534 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908537 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908541 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908544 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908547 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908565 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908569 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908571 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908574 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908578 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908580 2579 flags.go:64] FLAG: --kube-reserved="" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908585 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908588 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908591 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908593 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908596 2579 flags.go:64] FLAG: --lock-file="" Jul 14 18:10:01.371574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908599 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908601 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908604 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908610 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908613 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908616 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908618 2579 flags.go:64] FLAG: --logging-format="text" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908625 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908628 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908630 2579 flags.go:64] FLAG: --manifest-url="" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908633 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908638 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908641 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908645 2579 flags.go:64] FLAG: --max-pods="110" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908648 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908651 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908654 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908656 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908659 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908662 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908665 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908672 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908675 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908678 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 18:10:01.372256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908681 2579 flags.go:64] FLAG: --pod-cidr="" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908683 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908690 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908694 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908698 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908701 2579 flags.go:64] FLAG: --port="10250" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908704 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908706 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05b5f3ff729b23426" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908709 2579 flags.go:64] FLAG: --qos-reserved="" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908712 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908715 2579 flags.go:64] FLAG: --register-node="true" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908718 2579 flags.go:64] FLAG: --register-with-taints="" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908721 2579 flags.go:64] FLAG: --registry-burst="10" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908725 2579 flags.go:64] FLAG: --registry-qps="5" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908728 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908730 2579 flags.go:64] FLAG: --reserved-memory="" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908736 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908739 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908741 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908744 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908747 2579 flags.go:64] FLAG: --runonce="false" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908749 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908752 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908755 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908758 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 18:10:01.373025 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908761 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908763 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908766 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908769 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908771 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908774 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908777 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908780 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908783 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908785 2579 flags.go:64] FLAG: --system-cgroups="" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908790 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908795 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908798 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908800 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908804 2579 flags.go:64] FLAG: --tls-min-version="" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908807 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908810 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908812 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908815 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908818 2579 flags.go:64] FLAG: --v="2" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908822 2579 flags.go:64] FLAG: --version="false" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908826 2579 flags.go:64] FLAG: --vmodule="" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908830 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.908834 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908948 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:01.373811 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908952 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908955 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908957 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908960 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908963 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908965 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908968 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908970 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908973 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908975 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908978 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908980 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908983 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908985 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908988 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908990 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908993 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.908997 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909000 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909002 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:01.374561 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909005 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909007 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909010 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909012 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909015 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909017 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909020 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909022 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909025 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909028 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909031 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909034 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909037 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909039 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909042 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909044 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909047 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909049 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909052 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909054 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:01.375353 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909056 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909059 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909061 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909064 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909066 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909068 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909071 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909074 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909076 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909080 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909083 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909085 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909089 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909093 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909095 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909098 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909101 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909104 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909108 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:01.375906 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909110 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909113 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909115 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909120 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909122 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909126 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909129 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909132 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909135 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909146 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909149 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909151 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909154 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909156 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909159 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909161 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909164 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909167 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909169 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:01.376363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909172 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909174 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909195 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909199 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909202 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909204 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909207 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909209 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909211 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909214 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909217 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909219 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909222 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909224 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909227 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909229 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909233 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909236 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909239 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:01.376833 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.909241 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.909894 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.917607 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.917624 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.917644 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.917650 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917712 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917716 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917721 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917725 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917728 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917731 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917734 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917737 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:01.377323 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917740 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917743 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917745 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917748 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917750 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917753 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917756 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917759 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917762 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917765 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917768 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917770 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917773 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917775 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917778 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917781 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917784 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917786 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917789 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917792 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:01.377690 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917794 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917803 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917806 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917808 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917811 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917813 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917816 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917818 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917821 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917823 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917826 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917829 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917831 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917834 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917837 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917840 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917843 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917845 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917848 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917851 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:01.378189 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917853 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917856 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917859 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917862 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917864 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917867 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917870 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917875 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917878 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917881 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917885 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917888 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917891 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917894 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917899 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917902 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917905 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917907 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917910 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:01.378706 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917912 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917915 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917917 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917920 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917924 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917928 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917931 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917934 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917937 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917940 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917943 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917946 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917948 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917951 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917954 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917957 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917959 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917962 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917964 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:01.379154 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917967 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917971 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917973 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917976 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917978 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917981 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917984 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917986 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917989 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917992 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917995 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.917997 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918000 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.918005 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918126 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:01.379688 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918132 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918135 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918137 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918140 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918142 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918145 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918148 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918152 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918155 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918157 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918160 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918162 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918165 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918167 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918170 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918172 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918175 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918177 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918180 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918184 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:01.380058 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918186 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918189 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918191 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918194 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918196 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918199 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918202 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918205 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918208 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918211 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918213 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918216 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918218 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918221 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918224 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918226 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918229 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918231 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918234 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918238 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:01.380526 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918240 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918243 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918245 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918248 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918250 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918253 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918255 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918258 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918260 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918263 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918265 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918270 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918274 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918277 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918280 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918283 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918285 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918288 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918291 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:01.381057 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918295 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918298 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918301 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918304 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918307 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918309 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918312 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918316 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918319 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918321 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918324 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918326 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918330 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918333 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918336 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918338 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918341 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918343 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918346 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918348 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:01.381539 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918351 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918353 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918356 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918372 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918376 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918378 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918381 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918384 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918386 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918389 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918392 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918394 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918397 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918399 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918402 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918405 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918407 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918410 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:01.382032 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:00.918412 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.918417 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.918547 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.921126 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.922896 2579 server.go:1019] "Starting client certificate rotation" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.922994 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.923028 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.925713 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.929377 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.946184 2579 log.go:25] "Validated CRI v1 runtime API" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.948666 2579 log.go:25] "Validated CRI v1 image API" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.949607 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 18:10:01.382475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.956326 2579 fs.go:135] Filesystem UUIDs: map[52c3be0d-6354-4377-ba07-489d50fb6a00:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e543d30d-0ff4-4cbf-9e52-3a5bce0cb826:/dev/nvme0n1p4] Jul 14 18:10:01.382777 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.956343 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 18:10:01.382806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.961410 2579 manager.go:216] Machine: {Timestamp:2026-07-14 18:10:00.960021724 +0000 UTC m=+0.391877149 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3206308 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26bbdb6ccdcaeb4eb6eec14ae6c9ea SystemUUID:ec26bbdb-6ccd-caeb-4eb6-eec14ae6c9ea BootID:10bf8a7a-7aff-4326-bc02-4cdd29b091da Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:fa:00:92:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:fa:00:92:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:7f:4e:01:ad:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 18:10:01.382806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.961519 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 18:10:01.382806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.961670 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 18:10:01.382806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.965335 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 18:10:01.382806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.965364 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 18:10:01.382806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.966289 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 18:10:01.382806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.966299 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 18:10:01.382806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.966306 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.966396 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.967100 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.968993 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.973691 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.973812 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.973835 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.973844 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.973859 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.973868 2579 kubelet.go:398] "Adding apiserver pod source" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.973878 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.974184 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.974198 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.977050 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.979356 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.979382 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.979535 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.979547 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.979569 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.979574 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.979582 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.979587 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.980186 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.980193 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.980199 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.980206 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.980222 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.980229 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.982007 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.982017 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.986369 2579 server.go:1268] "Started kubelet" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.986518 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.986602 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.986664 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:00.988781 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:00.988787 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.988870 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.988922 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.991414 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 18:10:01.383041 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:00.994323 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.998679 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.998721 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:00.998794 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999459 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999461 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999490 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999592 2579 factory.go:55] Registering systemd factory Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999597 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999605 2579 factory.go:223] Registration of the systemd container factory successfully Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999607 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999827 2579 factory.go:153] Registering CRI-O factory Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999840 2579 factory.go:223] Registration of the crio container factory successfully Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999900 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999921 2579 factory.go:103] Registering Raw factory Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:00.999937 2579 manager.go:1196] Started watching for new ooms in manager Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.000313 2579 manager.go:349] Starting recovery of all containers Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.001398 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-146.ec2.internal.18c23a6366664aa9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-146.ec2.internal,UID:ip-10-0-129-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-146.ec2.internal,},FirstTimestamp:2026-07-14 18:10:00.986290857 +0000 UTC m=+0.418146304,LastTimestamp:2026-07-14 18:10:00.986290857 +0000 UTC m=+0.418146304,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-146.ec2.internal,}" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.005727 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.005759 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.010401 2579 manager.go:354] Recovery completed Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.017586 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.020471 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hvfnn" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.020487 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.020519 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.020532 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.021102 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.021102 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.021163 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.021197 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.021821 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.021849 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:01.384152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.021863 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.022982 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-146.ec2.internal.18c23a6368704ce4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-146.ec2.internal,UID:ip-10-0-129-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-146.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-146.ec2.internal,},FirstTimestamp:2026-07-14 18:10:01.02050122 +0000 UTC m=+0.452356647,LastTimestamp:2026-07-14 18:10:01.02050122 +0000 UTC m=+0.452356647,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-146.ec2.internal,}" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.023344 2579 policy_none.go:49] "None policy: Start" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.023357 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.023367 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.025337 2579 policy_none.go:47] "Start" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.031013 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hvfnn" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.033335 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-146.ec2.internal.18c23a636870aba5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-146.ec2.internal,UID:ip-10-0-129-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-146.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-146.ec2.internal,},FirstTimestamp:2026-07-14 18:10:01.020525477 +0000 UTC m=+0.452380900,LastTimestamp:2026-07-14 18:10:01.020525477 +0000 UTC m=+0.452380900,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-146.ec2.internal,}" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.035440 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-146.ec2.internal.18c23a636870d4d1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-146.ec2.internal,UID:ip-10-0-129-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-146.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-146.ec2.internal,},FirstTimestamp:2026-07-14 18:10:01.020536017 +0000 UTC m=+0.452391440,LastTimestamp:2026-07-14 18:10:01.020536017 +0000 UTC m=+0.452391440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-146.ec2.internal,}" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.098857 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.173153 2579 manager.go:337] "Starting Device Plugin manager" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.173384 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.173398 2579 server.go:87] "Starting device plugin registration server" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.173646 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.173656 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.173755 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.173869 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.173881 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.174409 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.174463 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.199446 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 18:10:01.384980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.200659 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.200689 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.200711 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.200742 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.202907 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.212815 2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-146.ec2.internal\" not found" node="ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.273945 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.274954 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.274980 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.274997 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.275032 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.287168 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.287191 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-146.ec2.internal\": node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.299073 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.301171 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"] Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.301244 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.303139 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.303165 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.303178 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.305538 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.305659 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.305688 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.306340 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.306368 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.306374 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.306384 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.306406 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.306419 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.308530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.308574 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.309250 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.309302 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.309313 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:01.385533 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.330185 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-146.ec2.internal\" not found" node="ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.386387 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.333511 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-146.ec2.internal\" not found" node="ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.399177 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.399150 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.401387 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.401369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0a09f4b6189732fc8c4662cf0c52d03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"c0a09f4b6189732fc8c4662cf0c52d03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.401468 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.401402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a09f4b6189732fc8c4662cf0c52d03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"c0a09f4b6189732fc8c4662cf0c52d03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.401468 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.401427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd0eec76c28a01b9629dd9799f6aa948-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"fd0eec76c28a01b9629dd9799f6aa948\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.499912 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.499852 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.502087 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.502067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0a09f4b6189732fc8c4662cf0c52d03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"c0a09f4b6189732fc8c4662cf0c52d03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.502166 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.502104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a09f4b6189732fc8c4662cf0c52d03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"c0a09f4b6189732fc8c4662cf0c52d03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.502166 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.502127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd0eec76c28a01b9629dd9799f6aa948-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"fd0eec76c28a01b9629dd9799f6aa948\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.502252 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.502170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0a09f4b6189732fc8c4662cf0c52d03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"c0a09f4b6189732fc8c4662cf0c52d03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.502252 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.502179 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd0eec76c28a01b9629dd9799f6aa948-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"fd0eec76c28a01b9629dd9799f6aa948\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.502252 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.502183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a09f4b6189732fc8c4662cf0c52d03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"c0a09f4b6189732fc8c4662cf0c52d03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.601290 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.601234 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.633756 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.633726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.635997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.635977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" Jul 14 18:10:01.702039 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.702003 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.806280 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.806189 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.907291 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:01.907251 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:01.922515 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.922491 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 18:10:01.923007 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:01.922681 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:10:02.007981 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:02.007946 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:02.035781 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.035743 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 18:05:01 +0000 UTC" deadline="2028-02-15 11:02:27.324443587 +0000 UTC" Jul 14 18:10:02.035781 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.035775 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13936h52m25.288671581s" Jul 14 18:10:02.037929 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.037911 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:02.039166 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.039148 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:02.039276 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.039186 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:02.039276 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.039198 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:02.109221 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:02.109140 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found" Jul 14 18:10:02.132448 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.132422 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:02.199688 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.199645 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" Jul 14 18:10:02.219906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.219874 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:10:02.221493 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.221477 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" Jul 14 18:10:02.241103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.241079 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:10:02.269497 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.269473 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:02.311034 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.311002 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:02.339540 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:02.338734 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a09f4b6189732fc8c4662cf0c52d03.slice/crio-b97589294059c0339992966048298d3e68d81085d0f5e905d7296da3adf43274 WatchSource:0}: Error finding container b97589294059c0339992966048298d3e68d81085d0f5e905d7296da3adf43274: Status 404 returned error can't find the container with id b97589294059c0339992966048298d3e68d81085d0f5e905d7296da3adf43274 Jul 14 18:10:02.340992 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:02.340963 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0eec76c28a01b9629dd9799f6aa948.slice/crio-38161adc9857e85e2973363c32ebb8b72de194d025a209f1a90d5b7593db81dd WatchSource:0}: Error finding container 38161adc9857e85e2973363c32ebb8b72de194d025a209f1a90d5b7593db81dd: Status 404 returned error can't find the container with id 38161adc9857e85e2973363c32ebb8b72de194d025a209f1a90d5b7593db81dd Jul 14 18:10:02.345147 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.345129 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:10:02.841728 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.841699 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:02.977712 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.977667 2579 apiserver.go:52] "Watching apiserver" Jul 14 18:10:02.985905 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.985880 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 18:10:02.987033 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.987012 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal","openshift-multus/multus-bgbpr","openshift-multus/multus-additional-cni-plugins-bvrtj","openshift-multus/network-metrics-daemon-t9k4k","openshift-network-diagnostics/network-check-target-wjkg4","openshift-network-operator/iptables-alerter-7llhw","openshift-ovn-kubernetes/ovnkube-node-z5gxk","kube-system/global-pull-secret-syncer-82p7g","kube-system/konnectivity-agent-j78bw","kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr","openshift-cluster-node-tuning-operator/tuned-lkppf","openshift-image-registry/node-ca-cckpq"] Jul 14 18:10:02.991251 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.991231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:02.993537 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.993501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bgbpr" Jul 14 18:10:02.995989 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.995966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:02.997911 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.997883 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 18:10:02.998017 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.997953 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 18:10:02.998316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.998303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:02.998388 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:02.998358 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:02.998758 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.998741 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 18:10:02.998833 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.998794 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 18:10:02.998833 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.998800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-57hlz\"" Jul 14 18:10:02.999822 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.999806 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 18:10:02.999882 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:02.999847 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 18:10:03.000607 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.000587 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 18:10:03.000735 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.000625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:03.000735 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.000687 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:03.001490 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.001465 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 18:10:03.001672 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.001515 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 18:10:03.001672 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.001531 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z58pp\"" Jul 14 18:10:03.001672 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.001467 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 18:10:03.001879 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.001861 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 18:10:03.003210 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.003191 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.004744 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.004727 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 18:10:03.004858 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.004841 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h4trv\"" Jul 14 18:10:03.005670 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.005640 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.005756 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.005718 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:03.006078 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.006062 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 18:10:03.008107 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.008089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:03.009160 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.009132 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2xqc\"" Jul 14 18:10:03.009265 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.009217 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:03.009449 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.009429 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:03.010868 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.010727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.010960 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.010900 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 18:10:03.011232 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.011210 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 18:10:03.011417 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.011398 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wgqnn\"" Jul 14 18:10:03.011936 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.011904 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 18:10:03.013481 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.013741 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp469\" (UniqueName: \"kubernetes.io/projected/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-kube-api-access-bp469\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.013741 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-cnibin\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.013932 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:03.013932 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-systemd-units\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.013932 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-ovn\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.013932 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-hostroot\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.013932 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-multus-certs\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.013932 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ad70cac-450f-45f9-ae27-44950b4ca824-cni-binary-copy\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-cnibin\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.013988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-os-release\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-conf-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw522\" (UniqueName: \"kubernetes.io/projected/2f03c086-3646-4ade-a3eb-9f7920147f1e-kube-api-access-rw522\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-cni-netd\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovnkube-config\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovn-node-metrics-cert\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovnkube-script-lib\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-netns\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-system-cni-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-socket-dir-parent\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-k8s-cni-cncf-io\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-kubelet\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766xm\" (UniqueName: \"kubernetes.io/projected/5ad70cac-450f-45f9-ae27-44950b4ca824-kube-api-access-766xm\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-os-release\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-cni-bin\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-cni-bin\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-cni-multus\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-daemon-config\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-slash\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-run-netns\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-etc-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-node-log\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.014919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-log-socket\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-env-overrides\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-etc-kubernetes\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-cni-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-kubelet\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-var-lib-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-system-cni-dir\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk467\" (UniqueName: \"kubernetes.io/projected/067d5347-e149-498f-b815-f682137d2b41-kube-api-access-nk467\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:03.015314 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.014866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-systemd\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.015892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.015877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.022099 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022078 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:03.022204 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022080 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 18:10:03.022204 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022124 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:03.022204 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022171 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 18:10:03.022204 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022185 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 18:10:03.022419 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022288 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 18:10:03.022419 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022382 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 18:10:03.022950 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022934 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5gnt5\"" Jul 14 18:10:03.023033 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022958 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hqzfk\"" Jul 14 18:10:03.023033 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.022985 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2k55\"" Jul 14 18:10:03.023141 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.023032 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 18:10:03.028682 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.028663 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:10:03.064834 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.064796 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wvd6q" Jul 14 18:10:03.083643 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.083617 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wvd6q" Jul 14 18:10:03.101132 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.101072 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 18:10:03.115318 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-systemd-units\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115385 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-ovn\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115385 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-device-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.115466 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-systemd-units\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115466 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khc9j\" (UniqueName: \"kubernetes.io/projected/75b932e3-25bd-4bb8-a092-4de99b1b0625-kube-api-access-khc9j\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.115466 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ad70cac-450f-45f9-ae27-44950b4ca824-cni-binary-copy\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.115594 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-ovn\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115594 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw522\" (UniqueName: \"kubernetes.io/projected/2f03c086-3646-4ade-a3eb-9f7920147f1e-kube-api-access-rw522\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.115670 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysctl-d\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.115670 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-cni-netd\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115670 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovnkube-config\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovn-node-metrics-cert\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovnkube-script-lib\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6325f19-b72d-491c-9eb6-69569b111a9f-agent-certs\") pod \"konnectivity-agent-j78bw\" (UID: \"e6325f19-b72d-491c-9eb6-69569b111a9f\") " pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:03.115920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-cni-netd\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.115920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-run\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.115920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjqbr\" (UniqueName: \"kubernetes.io/projected/246547c6-5965-4d98-a8b7-001cc8fe865b-kube-api-access-mjqbr\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.115920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-kubernetes\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.115920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-system-cni-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-socket-dir-parent\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.115992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-k8s-cni-cncf-io\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-os-release\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-sys\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/246547c6-5965-4d98-a8b7-001cc8fe865b-tmp\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-cni-bin\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-daemon-config\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-k8s-cni-cncf-io\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-var-lib-kubelet\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ad70cac-450f-45f9-ae27-44950b4ca824-cni-binary-copy\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75b932e3-25bd-4bb8-a092-4de99b1b0625-host\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.116316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovnkube-config\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovnkube-script-lib\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-cni-bin\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-slash\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116328 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-socket-dir-parent\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-etc-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-os-release\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-node-log\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-slash\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-node-log\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-env-overrides\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-etc-kubernetes\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-system-cni-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.117030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72jg\" (UniqueName: \"kubernetes.io/projected/ed3d66cc-2758-448e-be01-5f505d73249e-kube-api-access-f72jg\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78631d62-ebdf-43af-a039-83ba0683e841-dbus\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-etc-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-var-lib-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116593 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-etc-kubernetes\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-var-lib-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.116697 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-env-overrides\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.116831 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs podName:067d5347-e149-498f-b815-f682137d2b41 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:03.616809632 +0000 UTC m=+3.048665067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs") pod "network-metrics-daemon-t9k4k" (UID: "067d5347-e149-498f-b815-f682137d2b41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-daemon-config\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.117732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116889 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-openvswitch\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-hostroot\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.116936 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-multus-certs\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-hostroot\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-lib-modules\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117093 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-multus-certs\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3d66cc-2758-448e-be01-5f505d73249e-host-slash\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-cnibin\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-os-release\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-cnibin\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-conf-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-os-release\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.118409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75b932e3-25bd-4bb8-a092-4de99b1b0625-serviceca\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-conf-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-netns\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-run-netns\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-kubelet\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766xm\" (UniqueName: \"kubernetes.io/projected/5ad70cac-450f-45f9-ae27-44950b4ca824-kube-api-access-766xm\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117771 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-kubelet\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6325f19-b72d-491c-9eb6-69569b111a9f-konnectivity-ca\") pod \"konnectivity-agent-j78bw\" (UID: \"e6325f19-b72d-491c-9eb6-69569b111a9f\") " pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysconfig\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed3d66cc-2758-448e-be01-5f505d73249e-iptables-alerter-script\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.117977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-cni-bin\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-cni-bin\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-cni-multus\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-host\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118082 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f03c086-3646-4ade-a3eb-9f7920147f1e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-host-var-lib-cni-multus\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78631d62-ebdf-43af-a039-83ba0683e841-kubelet-config\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-modprobe-d\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-run-netns\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-log-socket\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-run-netns\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk467\" (UniqueName: \"kubernetes.io/projected/067d5347-e149-498f-b815-f682137d2b41-kube-api-access-nk467\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-log-socket\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.119472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-sys-fs\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdczf\" (UniqueName: \"kubernetes.io/projected/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-kube-api-access-qdczf\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-cni-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-kubelet\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-system-cni-dir\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-tuned\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad70cac-450f-45f9-ae27-44950b4ca824-multus-cni-dir\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-socket-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118468 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-host-kubelet\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-system-cni-dir\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-registration-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-systemd\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp469\" (UniqueName: \"kubernetes.io/projected/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-kube-api-access-bp469\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.119892 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-run-systemd\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.120292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-cnibin\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.120292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:03.120292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysctl-conf\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.120292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-systemd\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.120292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.118708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f03c086-3646-4ade-a3eb-9f7920147f1e-cnibin\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.120435 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.120421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-ovn-node-metrics-cert\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.131760 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.131739 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:03.131760 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.131762 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:03.131942 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.131775 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vb7fh for pod openshift-network-diagnostics/network-check-target-wjkg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:03.131942 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.131851 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh podName:74c6b78c-4275-40e0-aa17-e14f72c95cce nodeName:}" failed. No retries permitted until 2026-07-14 18:10:03.631830171 +0000 UTC m=+3.063685590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vb7fh" (UniqueName: "kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh") pod "network-check-target-wjkg4" (UID: "74c6b78c-4275-40e0-aa17-e14f72c95cce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:03.135126 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.135099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw522\" (UniqueName: \"kubernetes.io/projected/2f03c086-3646-4ade-a3eb-9f7920147f1e-kube-api-access-rw522\") pod \"multus-additional-cni-plugins-bvrtj\" (UID: \"2f03c086-3646-4ade-a3eb-9f7920147f1e\") " pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.135951 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.135934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-766xm\" (UniqueName: \"kubernetes.io/projected/5ad70cac-450f-45f9-ae27-44950b4ca824-kube-api-access-766xm\") pod \"multus-bgbpr\" (UID: \"5ad70cac-450f-45f9-ae27-44950b4ca824\") " pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.136859 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.136841 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk467\" (UniqueName: \"kubernetes.io/projected/067d5347-e149-498f-b815-f682137d2b41-kube-api-access-nk467\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:03.137031 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.137016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp469\" (UniqueName: \"kubernetes.io/projected/4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3-kube-api-access-bp469\") pod \"ovnkube-node-z5gxk\" (UID: \"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.206757 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.206706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"c0a09f4b6189732fc8c4662cf0c52d03","Type":"ContainerStarted","Data":"b97589294059c0339992966048298d3e68d81085d0f5e905d7296da3adf43274"} Jul 14 18:10:03.207711 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.207685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" event={"ID":"fd0eec76c28a01b9629dd9799f6aa948","Type":"ContainerStarted","Data":"38161adc9857e85e2973363c32ebb8b72de194d025a209f1a90d5b7593db81dd"} Jul 14 18:10:03.219010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.218984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysctl-conf\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-systemd\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219150 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-device-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.219150 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khc9j\" (UniqueName: \"kubernetes.io/projected/75b932e3-25bd-4bb8-a092-4de99b1b0625-kube-api-access-khc9j\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.219150 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-device-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.219281 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219196 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysctl-conf\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219281 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-systemd\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219281 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysctl-d\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219430 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6325f19-b72d-491c-9eb6-69569b111a9f-agent-certs\") pod \"konnectivity-agent-j78bw\" (UID: \"e6325f19-b72d-491c-9eb6-69569b111a9f\") " pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:03.219430 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-run\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219430 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqbr\" (UniqueName: \"kubernetes.io/projected/246547c6-5965-4d98-a8b7-001cc8fe865b-kube-api-access-mjqbr\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219430 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-kubernetes\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219430 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219406 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysctl-d\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-sys\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-run\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/246547c6-5965-4d98-a8b7-001cc8fe865b-tmp\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-kubernetes\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219522 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-sys\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-var-lib-kubelet\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75b932e3-25bd-4bb8-a092-4de99b1b0625-host\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-var-lib-kubelet\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.219664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72jg\" (UniqueName: \"kubernetes.io/projected/ed3d66cc-2758-448e-be01-5f505d73249e-kube-api-access-f72jg\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78631d62-ebdf-43af-a039-83ba0683e841-dbus\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75b932e3-25bd-4bb8-a092-4de99b1b0625-host\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-lib-modules\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3d66cc-2758-448e-be01-5f505d73249e-host-slash\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78631d62-ebdf-43af-a039-83ba0683e841-dbus\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75b932e3-25bd-4bb8-a092-4de99b1b0625-serviceca\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-lib-modules\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3d66cc-2758-448e-be01-5f505d73249e-host-slash\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.219894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6325f19-b72d-491c-9eb6-69569b111a9f-konnectivity-ca\") pod \"konnectivity-agent-j78bw\" (UID: \"e6325f19-b72d-491c-9eb6-69569b111a9f\") " pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:03.220040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysconfig\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed3d66cc-2758-448e-be01-5f505d73249e-iptables-alerter-script\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-host\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.220075 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78631d62-ebdf-43af-a039-83ba0683e841-kubelet-config\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-modprobe-d\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.220158 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret podName:78631d62-ebdf-43af-a039-83ba0683e841 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:03.720138473 +0000 UTC m=+3.151993898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret") pod "global-pull-secret-syncer-82p7g" (UID: "78631d62-ebdf-43af-a039-83ba0683e841") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-host\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-sys-fs\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdczf\" (UniqueName: \"kubernetes.io/projected/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-kube-api-access-qdczf\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-modprobe-d\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-sys-fs\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.220674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78631d62-ebdf-43af-a039-83ba0683e841-kubelet-config\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-sysconfig\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-tuned\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-socket-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-registration-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-registration-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-socket-dir\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75b932e3-25bd-4bb8-a092-4de99b1b0625-serviceca\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6325f19-b72d-491c-9eb6-69569b111a9f-konnectivity-ca\") pod \"konnectivity-agent-j78bw\" (UID: \"e6325f19-b72d-491c-9eb6-69569b111a9f\") " pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:03.221480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.220988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed3d66cc-2758-448e-be01-5f505d73249e-iptables-alerter-script\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.222135 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.222109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/246547c6-5965-4d98-a8b7-001cc8fe865b-tmp\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.222224 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.222168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6325f19-b72d-491c-9eb6-69569b111a9f-agent-certs\") pod \"konnectivity-agent-j78bw\" (UID: \"e6325f19-b72d-491c-9eb6-69569b111a9f\") " pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:03.222855 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.222836 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/246547c6-5965-4d98-a8b7-001cc8fe865b-etc-tuned\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.247987 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.247947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72jg\" (UniqueName: \"kubernetes.io/projected/ed3d66cc-2758-448e-be01-5f505d73249e-kube-api-access-f72jg\") pod \"iptables-alerter-7llhw\" (UID: \"ed3d66cc-2758-448e-be01-5f505d73249e\") " pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.248143 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.247947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdczf\" (UniqueName: \"kubernetes.io/projected/4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5-kube-api-access-qdczf\") pod \"aws-ebs-csi-driver-node-vsjdr\" (UID: \"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.248143 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.248097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khc9j\" (UniqueName: \"kubernetes.io/projected/75b932e3-25bd-4bb8-a092-4de99b1b0625-kube-api-access-khc9j\") pod \"node-ca-cckpq\" (UID: \"75b932e3-25bd-4bb8-a092-4de99b1b0625\") " pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.249128 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.249108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqbr\" (UniqueName: \"kubernetes.io/projected/246547c6-5965-4d98-a8b7-001cc8fe865b-kube-api-access-mjqbr\") pod \"tuned-lkppf\" (UID: \"246547c6-5965-4d98-a8b7-001cc8fe865b\") " pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.301312 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.301278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:03.307788 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.307765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bgbpr" Jul 14 18:10:03.312506 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:03.312473 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4f89fd_4d5d_4944_8c8f_2c1d16d46ef3.slice/crio-60ddca26b7c88ac191c1586e0c175c233308533e1f86d756324628f801c8713a WatchSource:0}: Error finding container 60ddca26b7c88ac191c1586e0c175c233308533e1f86d756324628f801c8713a: Status 404 returned error can't find the container with id 60ddca26b7c88ac191c1586e0c175c233308533e1f86d756324628f801c8713a Jul 14 18:10:03.315662 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.315621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" Jul 14 18:10:03.318436 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:03.318411 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad70cac_450f_45f9_ae27_44950b4ca824.slice/crio-38363f71875e661de4a641c018b76759acbba45af0cc9694452974762873dee9 WatchSource:0}: Error finding container 38363f71875e661de4a641c018b76759acbba45af0cc9694452974762873dee9: Status 404 returned error can't find the container with id 38363f71875e661de4a641c018b76759acbba45af0cc9694452974762873dee9 Jul 14 18:10:03.321216 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.321194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7llhw" Jul 14 18:10:03.326207 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:03.326167 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f03c086_3646_4ade_a3eb_9f7920147f1e.slice/crio-9d73c4e91567eb20a9d3c114484586b397f24c75d548aae99b9aaf5789d37f9d WatchSource:0}: Error finding container 9d73c4e91567eb20a9d3c114484586b397f24c75d548aae99b9aaf5789d37f9d: Status 404 returned error can't find the container with id 9d73c4e91567eb20a9d3c114484586b397f24c75d548aae99b9aaf5789d37f9d Jul 14 18:10:03.327622 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.327602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:03.333128 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:03.333095 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d66cc_2758_448e_be01_5f505d73249e.slice/crio-ab199ab1314261392ce49443dcd16bb1d534be8d1948ce0d55f6f8ab8146487a WatchSource:0}: Error finding container ab199ab1314261392ce49443dcd16bb1d534be8d1948ce0d55f6f8ab8146487a: Status 404 returned error can't find the container with id ab199ab1314261392ce49443dcd16bb1d534be8d1948ce0d55f6f8ab8146487a Jul 14 18:10:03.334306 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.334281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" Jul 14 18:10:03.339811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.339792 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lkppf" Jul 14 18:10:03.341767 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:03.341739 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6325f19_b72d_491c_9eb6_69569b111a9f.slice/crio-a4fdb5deb7de22a82faf6bea31ccc176cdaf41974bba342bf7bbd09805258652 WatchSource:0}: Error finding container a4fdb5deb7de22a82faf6bea31ccc176cdaf41974bba342bf7bbd09805258652: Status 404 returned error can't find the container with id a4fdb5deb7de22a82faf6bea31ccc176cdaf41974bba342bf7bbd09805258652 Jul 14 18:10:03.344072 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.344047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cckpq" Jul 14 18:10:03.347291 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:03.347266 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d74e7c5_c9c1_45dd_86e5_d3cf7aad52a5.slice/crio-29e0552eb40bc47e02c0c2fefdd93727fba95b472df788ee42ad548a49d47de1 WatchSource:0}: Error finding container 29e0552eb40bc47e02c0c2fefdd93727fba95b472df788ee42ad548a49d47de1: Status 404 returned error can't find the container with id 29e0552eb40bc47e02c0c2fefdd93727fba95b472df788ee42ad548a49d47de1 Jul 14 18:10:03.350562 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:03.350541 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246547c6_5965_4d98_a8b7_001cc8fe865b.slice/crio-6a6e869132df17ff13065b6b7d083f9d0865dd5e98d0c312be58bb1b8a7f2c97 WatchSource:0}: Error finding container 6a6e869132df17ff13065b6b7d083f9d0865dd5e98d0c312be58bb1b8a7f2c97: Status 404 returned error can't find the container with id 6a6e869132df17ff13065b6b7d083f9d0865dd5e98d0c312be58bb1b8a7f2c97 Jul 14 18:10:03.356205 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:03.356181 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b932e3_25bd_4bb8_a092_4de99b1b0625.slice/crio-3dd3e85dfa2e8a2642b136daa30037bfb0ffba00730266ce67f81fcfdc741466 WatchSource:0}: Error finding container 3dd3e85dfa2e8a2642b136daa30037bfb0ffba00730266ce67f81fcfdc741466: Status 404 returned error can't find the container with id 3dd3e85dfa2e8a2642b136daa30037bfb0ffba00730266ce67f81fcfdc741466 Jul 14 18:10:03.623678 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.623565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:03.623832 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.623711 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:03.623832 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.623779 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs podName:067d5347-e149-498f-b815-f682137d2b41 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:04.6237597 +0000 UTC m=+4.055615117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs") pod "network-metrics-daemon-t9k4k" (UID: "067d5347-e149-498f-b815-f682137d2b41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:03.724082 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.723961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:03.724082 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:03.724075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:03.724327 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.724145 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:03.724327 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.724176 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:03.724327 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.724191 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vb7fh for pod openshift-network-diagnostics/network-check-target-wjkg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:03.724327 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.724208 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:03.724327 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.724252 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh podName:74c6b78c-4275-40e0-aa17-e14f72c95cce nodeName:}" failed. No retries permitted until 2026-07-14 18:10:04.724233895 +0000 UTC m=+4.156089309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb7fh" (UniqueName: "kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh") pod "network-check-target-wjkg4" (UID: "74c6b78c-4275-40e0-aa17-e14f72c95cce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:03.724327 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:03.724284 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret podName:78631d62-ebdf-43af-a039-83ba0683e841 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:04.724263656 +0000 UTC m=+4.156119070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret") pod "global-pull-secret-syncer-82p7g" (UID: "78631d62-ebdf-43af-a039-83ba0683e841") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:04.086748 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.086701 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:05:03 +0000 UTC" deadline="2028-03-01 17:32:33.083934304 +0000 UTC" Jul 14 18:10:04.086748 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.086739 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14303h22m28.997201016s" Jul 14 18:10:04.211860 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.211762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lkppf" event={"ID":"246547c6-5965-4d98-a8b7-001cc8fe865b","Type":"ContainerStarted","Data":"6a6e869132df17ff13065b6b7d083f9d0865dd5e98d0c312be58bb1b8a7f2c97"} Jul 14 18:10:04.214975 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.214858 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" event={"ID":"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5","Type":"ContainerStarted","Data":"29e0552eb40bc47e02c0c2fefdd93727fba95b472df788ee42ad548a49d47de1"} Jul 14 18:10:04.221062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.221028 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j78bw" event={"ID":"e6325f19-b72d-491c-9eb6-69569b111a9f","Type":"ContainerStarted","Data":"a4fdb5deb7de22a82faf6bea31ccc176cdaf41974bba342bf7bbd09805258652"} Jul 14 18:10:04.223309 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.223276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7llhw" event={"ID":"ed3d66cc-2758-448e-be01-5f505d73249e","Type":"ContainerStarted","Data":"ab199ab1314261392ce49443dcd16bb1d534be8d1948ce0d55f6f8ab8146487a"} Jul 14 18:10:04.229011 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.226796 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"60ddca26b7c88ac191c1586e0c175c233308533e1f86d756324628f801c8713a"} Jul 14 18:10:04.234403 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.233699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cckpq" event={"ID":"75b932e3-25bd-4bb8-a092-4de99b1b0625","Type":"ContainerStarted","Data":"3dd3e85dfa2e8a2642b136daa30037bfb0ffba00730266ce67f81fcfdc741466"} Jul 14 18:10:04.238688 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.238639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" event={"ID":"2f03c086-3646-4ade-a3eb-9f7920147f1e","Type":"ContainerStarted","Data":"9d73c4e91567eb20a9d3c114484586b397f24c75d548aae99b9aaf5789d37f9d"} Jul 14 18:10:04.242773 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.242712 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bgbpr" event={"ID":"5ad70cac-450f-45f9-ae27-44950b4ca824","Type":"ContainerStarted","Data":"38363f71875e661de4a641c018b76759acbba45af0cc9694452974762873dee9"} Jul 14 18:10:04.634593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.634549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:04.634809 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:04.634734 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:04.634809 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:04.634807 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs podName:067d5347-e149-498f-b815-f682137d2b41 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:06.634787024 +0000 UTC m=+6.066642441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs") pod "network-metrics-daemon-t9k4k" (UID: "067d5347-e149-498f-b815-f682137d2b41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:04.735609 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.735479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:04.735609 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:04.735578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:04.735856 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:04.735726 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:04.735856 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:04.735791 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret podName:78631d62-ebdf-43af-a039-83ba0683e841 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:06.73577237 +0000 UTC m=+6.167627798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret") pod "global-pull-secret-syncer-82p7g" (UID: "78631d62-ebdf-43af-a039-83ba0683e841") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:04.736220 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:04.736199 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:04.736316 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:04.736231 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:04.736316 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:04.736245 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vb7fh for pod openshift-network-diagnostics/network-check-target-wjkg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:04.736316 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:04.736290 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh podName:74c6b78c-4275-40e0-aa17-e14f72c95cce nodeName:}" failed. No retries permitted until 2026-07-14 18:10:06.736276098 +0000 UTC m=+6.168131517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb7fh" (UniqueName: "kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh") pod "network-check-target-wjkg4" (UID: "74c6b78c-4275-40e0-aa17-e14f72c95cce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:05.087345 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:05.087226 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:05:03 +0000 UTC" deadline="2028-04-11 10:57:51.854942036 +0000 UTC" Jul 14 18:10:05.087345 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:05.087298 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15280h47m46.767648627s" Jul 14 18:10:05.202298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:05.201755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:05.202298 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:05.201892 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:05.202298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:05.201904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:05.202298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:05.201955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:05.202298 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:05.202021 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:05.202298 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:05.202104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:05.258118 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:05.258078 2579 generic.go:356] "Generic (PLEG): container finished" podID="c0a09f4b6189732fc8c4662cf0c52d03" containerID="072a4344c58ef43dc0646083a46ff8aa661a0c651317b397402a71e649ba3ea9" exitCode=0 Jul 14 18:10:05.258305 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:05.258138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"c0a09f4b6189732fc8c4662cf0c52d03","Type":"ContainerDied","Data":"072a4344c58ef43dc0646083a46ff8aa661a0c651317b397402a71e649ba3ea9"} Jul 14 18:10:06.266994 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:06.266239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"c0a09f4b6189732fc8c4662cf0c52d03","Type":"ContainerStarted","Data":"b7b0ad1efb97ee7a5f743f07cd3a3dc17b7d3934f9bfdfe4372f7e710791907e"} Jul 14 18:10:06.653832 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:06.653254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:06.653832 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:06.653487 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:06.653832 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:06.653562 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs podName:067d5347-e149-498f-b815-f682137d2b41 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:10.653540897 +0000 UTC m=+10.085396315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs") pod "network-metrics-daemon-t9k4k" (UID: "067d5347-e149-498f-b815-f682137d2b41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:06.753839 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:06.753796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:06.754027 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:06.753865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:06.754027 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:06.753974 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:06.754027 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:06.753989 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:06.754027 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:06.754006 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:06.754027 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:06.754019 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vb7fh for pod openshift-network-diagnostics/network-check-target-wjkg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:06.754302 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:06.754046 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret podName:78631d62-ebdf-43af-a039-83ba0683e841 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:10.754024685 +0000 UTC m=+10.185880121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret") pod "global-pull-secret-syncer-82p7g" (UID: "78631d62-ebdf-43af-a039-83ba0683e841") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:06.754302 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:06.754066 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh podName:74c6b78c-4275-40e0-aa17-e14f72c95cce nodeName:}" failed. No retries permitted until 2026-07-14 18:10:10.75405668 +0000 UTC m=+10.185912094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb7fh" (UniqueName: "kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh") pod "network-check-target-wjkg4" (UID: "74c6b78c-4275-40e0-aa17-e14f72c95cce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:07.201484 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.201432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:07.201484 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.201476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:07.201768 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:07.201587 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:07.201768 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.201434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:07.201768 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:07.201714 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:07.201919 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:07.201804 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:07.399113 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.399042 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" podStartSLOduration=5.399018031 podStartE2EDuration="5.399018031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:06.285826076 +0000 UTC m=+5.717681515" watchObservedRunningTime="2026-07-14 18:10:07.399018031 +0000 UTC m=+6.830873469" Jul 14 18:10:07.399616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.399516 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-84xwb"] Jul 14 18:10:07.403115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.402592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.413933 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.413909 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 18:10:07.414170 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.414153 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 18:10:07.414388 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.414371 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89hm8\"" Jul 14 18:10:07.459411 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.459305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-tmp-dir\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.459593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.459423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qx29\" (UniqueName: \"kubernetes.io/projected/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-kube-api-access-7qx29\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.459593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.459463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-hosts-file\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.561160 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.560708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qx29\" (UniqueName: \"kubernetes.io/projected/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-kube-api-access-7qx29\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.561160 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.560779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-hosts-file\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.561160 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.560821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-tmp-dir\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.561160 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.560921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-hosts-file\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.561160 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.561167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-tmp-dir\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.578479 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.578411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qx29\" (UniqueName: \"kubernetes.io/projected/9c17b1f7-00f3-4d65-941f-16eb0c2e9d24-kube-api-access-7qx29\") pod \"node-resolver-84xwb\" (UID: \"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24\") " pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:07.717286 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:07.717007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84xwb" Jul 14 18:10:09.201873 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:09.201709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:09.201873 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:09.201728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:09.202563 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:09.201894 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:09.202563 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:09.201969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:09.202563 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:09.202109 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:09.202563 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:09.202187 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:10.690660 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:10.690013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:10.690660 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:10.690217 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:10.690660 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:10.690291 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs podName:067d5347-e149-498f-b815-f682137d2b41 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:18.69027067 +0000 UTC m=+18.122126092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs") pod "network-metrics-daemon-t9k4k" (UID: "067d5347-e149-498f-b815-f682137d2b41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:10.791390 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:10.791057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:10.791390 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:10.791132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:10.791390 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:10.791177 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:10.791390 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:10.791254 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:10.791390 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:10.791271 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:10.791390 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:10.791288 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret podName:78631d62-ebdf-43af-a039-83ba0683e841 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:18.791259333 +0000 UTC m=+18.223114766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret") pod "global-pull-secret-syncer-82p7g" (UID: "78631d62-ebdf-43af-a039-83ba0683e841") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:10.791390 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:10.791319 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vb7fh for pod openshift-network-diagnostics/network-check-target-wjkg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:10.791390 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:10.791361 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh podName:74c6b78c-4275-40e0-aa17-e14f72c95cce nodeName:}" failed. No retries permitted until 2026-07-14 18:10:18.791349782 +0000 UTC m=+18.223205195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb7fh" (UniqueName: "kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh") pod "network-check-target-wjkg4" (UID: "74c6b78c-4275-40e0-aa17-e14f72c95cce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:11.203739 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:11.201714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:11.203739 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:11.201852 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:11.203739 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:11.202248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:11.203739 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:11.202352 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:11.203739 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:11.202502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:11.203739 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:11.202621 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:13.203194 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:13.201934 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:13.203194 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:13.202054 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:13.203194 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:13.202447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:13.203194 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:13.202538 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:13.203194 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:13.202625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:13.203194 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:13.202709 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:15.201089 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:15.201054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:15.201474 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:15.201067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:15.201474 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:15.201163 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:15.201474 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:15.201213 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:15.201474 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:15.201245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:15.201474 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:15.201292 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:17.201321 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:17.201281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:17.201823 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:17.201281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:17.201823 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:17.201441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:17.201823 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:17.201410 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:17.201823 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:17.201548 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:17.201823 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:17.201649 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:18.746603 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:18.746565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:18.747034 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:18.746662 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:18.747034 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:18.746747 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs podName:067d5347-e149-498f-b815-f682137d2b41 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:34.746725816 +0000 UTC m=+34.178581228 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs") pod "network-metrics-daemon-t9k4k" (UID: "067d5347-e149-498f-b815-f682137d2b41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:18.847131 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:18.847096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:18.847328 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:18.847183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:18.847328 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:18.847288 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:18.847328 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:18.847320 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:18.847474 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:18.847333 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vb7fh for pod openshift-network-diagnostics/network-check-target-wjkg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:18.847474 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:18.847296 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:18.847474 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:18.847399 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh podName:74c6b78c-4275-40e0-aa17-e14f72c95cce nodeName:}" failed. No retries permitted until 2026-07-14 18:10:34.847380775 +0000 UTC m=+34.279236186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb7fh" (UniqueName: "kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh") pod "network-check-target-wjkg4" (UID: "74c6b78c-4275-40e0-aa17-e14f72c95cce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:18.847474 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:18.847469 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret podName:78631d62-ebdf-43af-a039-83ba0683e841 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:34.847449002 +0000 UTC m=+34.279304424 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret") pod "global-pull-secret-syncer-82p7g" (UID: "78631d62-ebdf-43af-a039-83ba0683e841") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:19.201675 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:19.201595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:19.201675 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:19.201619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:19.201866 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:19.201608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:19.201866 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:19.201725 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:19.201866 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:19.201837 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:19.202017 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:19.201919 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:21.202378 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:21.202349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:21.202825 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:21.202426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:21.202825 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:21.202432 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:21.202825 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:21.202571 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:21.202825 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:21.202579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:21.202825 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:21.202673 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:23.201750 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:23.201717 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:23.201750 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:23.201733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:23.202284 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:23.201819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:23.202284 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:23.201844 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:23.202284 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:23.201888 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:23.202284 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:23.201978 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:23.217153 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:23.217128 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c17b1f7_00f3_4d65_941f_16eb0c2e9d24.slice/crio-dff22c2308df5f0c26c6e3a709583c8ad8fc45edc0d217ba8b0fdd918f6c0f42 WatchSource:0}: Error finding container dff22c2308df5f0c26c6e3a709583c8ad8fc45edc0d217ba8b0fdd918f6c0f42: Status 404 returned error can't find the container with id dff22c2308df5f0c26c6e3a709583c8ad8fc45edc0d217ba8b0fdd918f6c0f42 Jul 14 18:10:23.320511 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:23.320282 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84xwb" event={"ID":"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24","Type":"ContainerStarted","Data":"dff22c2308df5f0c26c6e3a709583c8ad8fc45edc0d217ba8b0fdd918f6c0f42"} Jul 14 18:10:24.323640 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.323397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84xwb" event={"ID":"9c17b1f7-00f3-4d65-941f-16eb0c2e9d24","Type":"ContainerStarted","Data":"c0376bfb9fdd8439864986862dc16e47d31aef60500e7a1985b236f18561b1d5"} Jul 14 18:10:24.324822 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.324794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cckpq" event={"ID":"75b932e3-25bd-4bb8-a092-4de99b1b0625","Type":"ContainerStarted","Data":"d2b4a0864bb5d7a1b3954965e98c1d3fe6c12819a412e38641224f65a6c45794"} Jul 14 18:10:24.326107 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.326087 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f03c086-3646-4ade-a3eb-9f7920147f1e" containerID="320819e036dc0d2c57f72c57dae6e3c42c4a7bb72e97ebd82fb96a1cfe20e6c6" exitCode=0 Jul 14 18:10:24.326180 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.326147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" event={"ID":"2f03c086-3646-4ade-a3eb-9f7920147f1e","Type":"ContainerDied","Data":"320819e036dc0d2c57f72c57dae6e3c42c4a7bb72e97ebd82fb96a1cfe20e6c6"} Jul 14 18:10:24.327598 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.327581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bgbpr" event={"ID":"5ad70cac-450f-45f9-ae27-44950b4ca824","Type":"ContainerStarted","Data":"5d8486fd637db07e03734c01694f50524ca36f9039014b09b81fa16d99177cdf"} Jul 14 18:10:24.329057 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.329010 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" event={"ID":"fd0eec76c28a01b9629dd9799f6aa948","Type":"ContainerStarted","Data":"64b32ca68728d0796a0521dbb9a4e8fa894260e18664c77139366b5078372385"} Jul 14 18:10:24.330494 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.330475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lkppf" event={"ID":"246547c6-5965-4d98-a8b7-001cc8fe865b","Type":"ContainerStarted","Data":"f741c9d796ba1a67da0da762e5f33899bf8ea60c589f94fb5dd2ba01ededaeb8"} Jul 14 18:10:24.331867 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.331849 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" event={"ID":"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5","Type":"ContainerStarted","Data":"bd9dba75fe2e90a865b5ab7f1bfdc2e801cad4471a399b9bacf9e312be42e78d"} Jul 14 18:10:24.332923 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.332904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j78bw" event={"ID":"e6325f19-b72d-491c-9eb6-69569b111a9f","Type":"ContainerStarted","Data":"96d2541c898f54eaab0bce8eae671a15e939047acca385fad5a4f5de4c9dfefd"} Jul 14 18:10:24.335792 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.335777 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:10:24.336097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.336078 2579 generic.go:356] "Generic (PLEG): container finished" podID="4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3" containerID="b0b4a9589b55e9bd5981fe78d442ce954ee2ca833564ca28f24074fa9712cbfe" exitCode=1 Jul 14 18:10:24.336155 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.336106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"eb71583bbbfbd180b8284d0d780623bbb0b04f78c22ba3a23b66a32e24fa07ea"} Jul 14 18:10:24.336155 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.336148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"bd897612911b35cc04a90f4059eb4a6bc4dc0e680b56567879da386cf3e8ef46"} Jul 14 18:10:24.336235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.336178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"cdf4cc5134175cf0930585b1e6dc53bc72ec150ae7d55145311340903f80599e"} Jul 14 18:10:24.336235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.336206 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"7b5c7786065e1066c73c1f8b3a9aa0afe524d951d48db6705899f1149f4b4024"} Jul 14 18:10:24.336235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.336232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerDied","Data":"b0b4a9589b55e9bd5981fe78d442ce954ee2ca833564ca28f24074fa9712cbfe"} Jul 14 18:10:24.336321 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.336259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"6129d3900aef154c424b9c12e44c10a2eefde2509df90072c40268bfd3173841"} Jul 14 18:10:24.342040 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.341994 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-84xwb" podStartSLOduration=17.341978707 podStartE2EDuration="17.341978707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:24.341534921 +0000 UTC m=+23.773390369" watchObservedRunningTime="2026-07-14 18:10:24.341978707 +0000 UTC m=+23.773834142" Jul 14 18:10:24.362635 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.362590 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cckpq" podStartSLOduration=2.506300761 podStartE2EDuration="22.362578097s" totalImagesPullingTime="19.856277336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:24.362329401 +0000 UTC m=+23.794184899" watchObservedRunningTime="2026-07-14 18:10:24.362578097 +0000 UTC m=+23.794433531" Jul 14 18:10:24.407935 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.407886 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" podStartSLOduration=21.407872869 podStartE2EDuration="21.407872869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:24.380439459 +0000 UTC m=+23.812294893" watchObservedRunningTime="2026-07-14 18:10:24.407872869 +0000 UTC m=+23.839728299" Jul 14 18:10:24.408113 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.408096 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bgbpr" podStartSLOduration=2.454191684 podStartE2EDuration="22.408090277s" totalImagesPullingTime="19.953898593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:24.407663875 +0000 UTC m=+23.839519311" watchObservedRunningTime="2026-07-14 18:10:24.408090277 +0000 UTC m=+23.839945711" Jul 14 18:10:24.455912 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.455859 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j78bw" podStartSLOduration=2.585591162 podStartE2EDuration="22.455842933s" totalImagesPullingTime="19.870251771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:24.455734982 +0000 UTC m=+23.887590427" watchObservedRunningTime="2026-07-14 18:10:24.455842933 +0000 UTC m=+23.887698369" Jul 14 18:10:24.477275 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.477177 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lkppf" podStartSLOduration=2.606307879 podStartE2EDuration="22.477155975s" totalImagesPullingTime="19.870848096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:24.47688566 +0000 UTC m=+23.908741094" watchObservedRunningTime="2026-07-14 18:10:24.477155975 +0000 UTC m=+23.909011410" Jul 14 18:10:24.789291 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:24.789269 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 18:10:25.185173 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:25.185008 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T18:10:24.789285998Z","UUID":"d69d6e3f-afe2-4682-8e46-c908ff2801cf","Handler":null,"Name":"","Endpoint":""} Jul 14 18:10:25.186452 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:25.186427 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 18:10:25.186596 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:25.186462 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 18:10:25.201227 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:25.201203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:25.201382 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:25.201211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:25.201382 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:25.201322 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:25.201382 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:25.201211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:25.201580 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:25.201383 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:25.201580 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:25.201442 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:25.341112 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:25.341065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" event={"ID":"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5","Type":"ContainerStarted","Data":"09774f89eab1e8f97cd6f629d3f82ebb67eaa8f36f0233b22220efe1e6f3fbfd"} Jul 14 18:10:25.342896 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:25.342855 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7llhw" event={"ID":"ed3d66cc-2758-448e-be01-5f505d73249e","Type":"ContainerStarted","Data":"5cf9995528eaa35bc32f576752785f70ab0b4fee23eae994bf8649d21ecf6f4b"} Jul 14 18:10:26.347570 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:26.347239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" event={"ID":"4d74e7c5-c9c1-45dd-86e5-d3cf7aad52a5","Type":"ContainerStarted","Data":"2c7c5504bed3d32cb56ed9f69e318eed0e4ea0592d7fc695d57a5491b96cd1bb"} Jul 14 18:10:26.350185 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:26.350165 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:10:26.350594 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:26.350574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"16bc6c3ddf90a8206b35b5d8f375f361acbf4ef7f50d3b980a7d87182030611c"} Jul 14 18:10:26.370782 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:26.370735 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7llhw" podStartSLOduration=4.493050815 podStartE2EDuration="24.370707062s" totalImagesPullingTime="19.877656247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:25.362543933 +0000 UTC m=+24.794399364" watchObservedRunningTime="2026-07-14 18:10:26.370707062 +0000 UTC m=+25.802562495" Jul 14 18:10:27.201052 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:27.200975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:27.201265 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:27.200975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:27.201265 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:27.201117 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:27.201265 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:27.201024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:27.201265 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:27.201207 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:27.201429 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:27.201299 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:28.359827 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.359605 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:10:28.360475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.360141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"2c6ecaf160ce4c76838a9fa25ee0bb06f63de61ede1aa2d11f84ae21c3dfdc88"} Jul 14 18:10:28.360475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.360408 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:28.360475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.360467 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:28.360806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.360783 2579 scope.go:117] "RemoveContainer" containerID="b0b4a9589b55e9bd5981fe78d442ce954ee2ca833564ca28f24074fa9712cbfe" Jul 14 18:10:28.378575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.378535 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:28.378900 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.378885 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:28.400615 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.400571 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsjdr" podStartSLOduration=4.342185815 podStartE2EDuration="26.400536553s" totalImagesPullingTime="22.058350738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:26.371256739 +0000 UTC m=+25.803112171" watchObservedRunningTime="2026-07-14 18:10:28.400536553 +0000 UTC m=+27.832391985" Jul 14 18:10:28.440178 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.440149 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:28.440976 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:28.440956 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:29.201933 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.201900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:29.202106 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:29.202031 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:29.202106 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.201904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:29.202228 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.201900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:29.202228 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:29.202127 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:29.202332 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:29.202252 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:29.366567 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.366534 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:10:29.367212 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.367181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" event={"ID":"4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3","Type":"ContainerStarted","Data":"c5dc4e42f17171a1a23b581a1d9c87e2d825c419e1dddf7be7212a30e080a8f2"} Jul 14 18:10:29.367332 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.367266 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 18:10:29.419960 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.419917 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" podStartSLOduration=7.424253777 podStartE2EDuration="27.419901878s" totalImagesPullingTime="19.995648101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:29.419798969 +0000 UTC m=+28.851654413" watchObservedRunningTime="2026-07-14 18:10:29.419901878 +0000 UTC m=+28.851757312" Jul 14 18:10:29.944176 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.944140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:29.944747 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:29.944725 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j78bw" Jul 14 18:10:30.046531 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.046492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:30.148167 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.148123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:30.370278 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.370241 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 18:10:30.526615 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.526583 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:10:30.670912 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.670819 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wjkg4"] Jul 14 18:10:30.671068 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.670936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:30.671068 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:30.671038 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:30.674078 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.673367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t9k4k"] Jul 14 18:10:30.674078 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.673496 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82p7g"] Jul 14 18:10:30.674078 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.673614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:30.674078 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:30.673735 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:30.674078 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:30.673816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:30.674078 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:30.673930 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:31.373843 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:31.373608 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f03c086-3646-4ade-a3eb-9f7920147f1e" containerID="58b16aa4088806c56427cd92613c0ba7c304a7894d2b1ae9c258bb1cac7c7af0" exitCode=0 Jul 14 18:10:31.373843 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:31.373687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" event={"ID":"2f03c086-3646-4ade-a3eb-9f7920147f1e","Type":"ContainerDied","Data":"58b16aa4088806c56427cd92613c0ba7c304a7894d2b1ae9c258bb1cac7c7af0"} Jul 14 18:10:32.037868 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:32.037835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:32.101132 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:32.101103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:32.172587 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:32.172547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:32.201734 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:32.201701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:32.201880 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:32.201742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:32.201880 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:32.201740 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:32.201880 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:32.201819 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:32.202039 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:32.201917 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:32.202039 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:32.201989 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:33.380484 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:33.380451 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f03c086-3646-4ade-a3eb-9f7920147f1e" containerID="1c628ccaa6a2589374b07f8357b5a94ed5e98a9fef7dc1902b2f0a7ebd60a362" exitCode=0 Jul 14 18:10:33.381070 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:33.380491 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" event={"ID":"2f03c086-3646-4ade-a3eb-9f7920147f1e","Type":"ContainerDied","Data":"1c628ccaa6a2589374b07f8357b5a94ed5e98a9fef7dc1902b2f0a7ebd60a362"} Jul 14 18:10:34.200914 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:34.200882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:34.201066 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:34.200888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:34.201066 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:34.201018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:34.201177 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.200990 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t9k4k" podUID="067d5347-e149-498f-b815-f682137d2b41" Jul 14 18:10:34.201177 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.201086 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82p7g" podUID="78631d62-ebdf-43af-a039-83ba0683e841" Jul 14 18:10:34.201270 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.201203 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjkg4" podUID="74c6b78c-4275-40e0-aa17-e14f72c95cce" Jul 14 18:10:34.760629 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:34.760528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:34.760966 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.760671 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:34.760966 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.760735 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs podName:067d5347-e149-498f-b815-f682137d2b41 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:06.760718071 +0000 UTC m=+66.192573484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs") pod "network-metrics-daemon-t9k4k" (UID: "067d5347-e149-498f-b815-f682137d2b41") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:34.861575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:34.861523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:34.861723 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:34.861606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:34.861723 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.861708 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:34.861783 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.861763 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret podName:78631d62-ebdf-43af-a039-83ba0683e841 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:06.861749975 +0000 UTC m=+66.293605391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret") pod "global-pull-secret-syncer-82p7g" (UID: "78631d62-ebdf-43af-a039-83ba0683e841") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:34.861842 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.861713 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:34.861842 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.861810 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:34.861842 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.861823 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vb7fh for pod openshift-network-diagnostics/network-check-target-wjkg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:34.861923 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:34.861869 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh podName:74c6b78c-4275-40e0-aa17-e14f72c95cce nodeName:}" failed. No retries permitted until 2026-07-14 18:11:06.861854796 +0000 UTC m=+66.293710208 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb7fh" (UniqueName: "kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh") pod "network-check-target-wjkg4" (UID: "74c6b78c-4275-40e0-aa17-e14f72c95cce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:35.647605 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.647532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:35.746978 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.746931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:35.859939 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.859912 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeReady" Jul 14 18:10:35.860330 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.860049 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 18:10:35.930256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.930227 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-658f887d4f-cfwbv"] Jul 14 18:10:35.942768 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.942743 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qxkkc"] Jul 14 18:10:35.942907 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.942863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:35.948745 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.948719 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 18:10:35.949514 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.949332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 18:10:35.949514 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.949428 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 18:10:35.950193 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.950174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kfgwx\"" Jul 14 18:10:35.954310 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.954292 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lg979"] Jul 14 18:10:35.954479 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.954464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxkkc" Jul 14 18:10:35.955571 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.955526 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 18:10:35.957870 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.957851 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hhvn4\"" Jul 14 18:10:35.958030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.957856 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 18:10:35.958030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.957946 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 18:10:35.958030 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.957977 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 18:10:35.968847 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.968828 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxkkc"] Jul 14 18:10:35.968938 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.968915 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lg979" Jul 14 18:10:35.972226 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.972089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 18:10:35.972226 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.972148 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 18:10:35.972398 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.972333 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lg979"] Jul 14 18:10:35.972398 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.972341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zklb7\"" Jul 14 18:10:35.976868 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:35.976846 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-658f887d4f-cfwbv"] Jul 14 18:10:36.060960 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.060875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:36.067396 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-image-registry-private-configuration\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.067472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-bound-sa-token\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.067472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9d7\" (UniqueName: \"kubernetes.io/projected/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-kube-api-access-bb9d7\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.067472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-config-volume\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.067590 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.067590 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-registry-certificates\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.067650 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.067650 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-installation-pull-secrets\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.067712 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-tmp-dir\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.067712 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d206d5-a34a-4da9-8d97-2ccb5c1c707d-cert\") pod \"ingress-canary-qxkkc\" (UID: \"88d206d5-a34a-4da9-8d97-2ccb5c1c707d\") " pod="openshift-ingress-canary/ingress-canary-qxkkc" Jul 14 18:10:36.067770 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-trusted-ca\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.067770 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-kube-api-access-48fsv\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.067832 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21df3e31-0497-4f60-b03c-edf3cf852737-ca-trust-extracted\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.067863 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.067831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcwvh\" (UniqueName: \"kubernetes.io/projected/88d206d5-a34a-4da9-8d97-2ccb5c1c707d-kube-api-access-rcwvh\") pod \"ingress-canary-qxkkc\" (UID: \"88d206d5-a34a-4da9-8d97-2ccb5c1c707d\") " pod="openshift-ingress-canary/ingress-canary-qxkkc" Jul 14 18:10:36.157087 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.157060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:36.168142 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-tmp-dir\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.168233 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d206d5-a34a-4da9-8d97-2ccb5c1c707d-cert\") pod \"ingress-canary-qxkkc\" (UID: \"88d206d5-a34a-4da9-8d97-2ccb5c1c707d\") " pod="openshift-ingress-canary/ingress-canary-qxkkc" Jul 14 18:10:36.168233 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-trusted-ca\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.168233 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-kube-api-access-48fsv\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.168374 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21df3e31-0497-4f60-b03c-edf3cf852737-ca-trust-extracted\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.168374 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcwvh\" (UniqueName: \"kubernetes.io/projected/88d206d5-a34a-4da9-8d97-2ccb5c1c707d-kube-api-access-rcwvh\") pod \"ingress-canary-qxkkc\" (UID: \"88d206d5-a34a-4da9-8d97-2ccb5c1c707d\") " pod="openshift-ingress-canary/ingress-canary-qxkkc" Jul 14 18:10:36.168374 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-image-registry-private-configuration\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.168374 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-bound-sa-token\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.168374 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9d7\" (UniqueName: \"kubernetes.io/projected/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-kube-api-access-bb9d7\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-config-volume\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-registry-certificates\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168503 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-tmp-dir\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-installation-pull-secrets\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.168650 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.168666 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f887d4f-cfwbv: secret "image-registry-tls" not found Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.168683 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.168730 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls podName:21df3e31-0497-4f60-b03c-edf3cf852737 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:36.668709487 +0000 UTC m=+36.100564902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls") pod "image-registry-658f887d4f-cfwbv" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737") : secret "image-registry-tls" not found Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.168748 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls podName:8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:36.668739049 +0000 UTC m=+36.100594460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls") pod "dns-default-lg979" (UID: "8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6") : secret "dns-default-metrics-tls" not found Jul 14 18:10:36.168906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.168896 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21df3e31-0497-4f60-b03c-edf3cf852737-ca-trust-extracted\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.169617 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.169137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-config-volume\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.169617 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.169232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-registry-certificates\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.169716 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.169650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-trusted-ca\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.172684 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.172540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-image-registry-private-configuration\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.172780 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.172581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-installation-pull-secrets\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.172780 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.172614 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d206d5-a34a-4da9-8d97-2ccb5c1c707d-cert\") pod \"ingress-canary-qxkkc\" (UID: \"88d206d5-a34a-4da9-8d97-2ccb5c1c707d\") " pod="openshift-ingress-canary/ingress-canary-qxkkc" Jul 14 18:10:36.180268 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.180241 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9d7\" (UniqueName: \"kubernetes.io/projected/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-kube-api-access-bb9d7\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.180384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.180283 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-kube-api-access-48fsv\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.180524 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.180508 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-bound-sa-token\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.180867 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.180846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcwvh\" (UniqueName: \"kubernetes.io/projected/88d206d5-a34a-4da9-8d97-2ccb5c1c707d-kube-api-access-rcwvh\") pod \"ingress-canary-qxkkc\" (UID: \"88d206d5-a34a-4da9-8d97-2ccb5c1c707d\") " pod="openshift-ingress-canary/ingress-canary-qxkkc" Jul 14 18:10:36.201200 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.201171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:10:36.201200 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.201198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:10:36.201383 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.201198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:10:36.203729 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.203702 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:10:36.203860 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.203779 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5vgxh\"" Jul 14 18:10:36.204020 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.204000 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jkhfz\"" Jul 14 18:10:36.204094 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.204055 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:10:36.204152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.204090 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:10:36.204152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.204115 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:10:36.263631 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.263597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxkkc" Jul 14 18:10:36.346919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.346821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:36.392294 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.392255 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f03c086-3646-4ade-a3eb-9f7920147f1e" containerID="cd965af41098f4370948d85096ddd8d50e61e7a69d56d0d45f61a240f474cc0e" exitCode=0 Jul 14 18:10:36.392507 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.392325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" event={"ID":"2f03c086-3646-4ade-a3eb-9f7920147f1e","Type":"ContainerDied","Data":"cd965af41098f4370948d85096ddd8d50e61e7a69d56d0d45f61a240f474cc0e"} Jul 14 18:10:36.458887 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.458858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:36.468231 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.468202 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxkkc"] Jul 14 18:10:36.472231 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:36.472195 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d206d5_a34a_4da9_8d97_2ccb5c1c707d.slice/crio-343c3da48658f540be3e47000ec4c4f6863d42b7ffcfda07dabc27a32a3f9640 WatchSource:0}: Error finding container 343c3da48658f540be3e47000ec4c4f6863d42b7ffcfda07dabc27a32a3f9640: Status 404 returned error can't find the container with id 343c3da48658f540be3e47000ec4c4f6863d42b7ffcfda07dabc27a32a3f9640 Jul 14 18:10:36.571281 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.571250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:36.672274 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.672184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:36.673243 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.673218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:36.673365 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:36.673253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:36.673413 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.673367 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:36.673413 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.673382 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f887d4f-cfwbv: secret "image-registry-tls" not found Jul 14 18:10:36.673496 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.673416 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:36.673496 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.673462 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls podName:21df3e31-0497-4f60-b03c-edf3cf852737 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.673447221 +0000 UTC m=+37.105302636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls") pod "image-registry-658f887d4f-cfwbv" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737") : secret "image-registry-tls" not found Jul 14 18:10:36.673496 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:36.673476 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls podName:8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.673469955 +0000 UTC m=+37.105325366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls") pod "dns-default-lg979" (UID: "8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6") : secret "dns-default-metrics-tls" not found Jul 14 18:10:37.320967 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:37.320932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:37.396200 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:37.396155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxkkc" event={"ID":"88d206d5-a34a-4da9-8d97-2ccb5c1c707d","Type":"ContainerStarted","Data":"343c3da48658f540be3e47000ec4c4f6863d42b7ffcfda07dabc27a32a3f9640"} Jul 14 18:10:37.409826 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:37.409789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:37.682676 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:37.682585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:37.682676 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:37.682643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:37.682910 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:37.682748 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:37.682910 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:37.682812 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls podName:8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:39.682797068 +0000 UTC m=+39.114652480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls") pod "dns-default-lg979" (UID: "8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6") : secret "dns-default-metrics-tls" not found Jul 14 18:10:37.682910 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:37.682753 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:37.682910 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:37.682856 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f887d4f-cfwbv: secret "image-registry-tls" not found Jul 14 18:10:37.683084 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:37.682917 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls podName:21df3e31-0497-4f60-b03c-edf3cf852737 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:39.682897872 +0000 UTC m=+39.114753299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls") pod "image-registry-658f887d4f-cfwbv" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737") : secret "image-registry-tls" not found Jul 14 18:10:37.702803 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:37.702718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:37.803436 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:37.803403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:39.061702 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.061671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:39.074019 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.073989 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt"] Jul 14 18:10:39.086805 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.086781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" Jul 14 18:10:39.091766 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.091739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt"] Jul 14 18:10:39.097987 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.097963 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-brzqc\"" Jul 14 18:10:39.101720 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.101695 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:39.102155 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.102133 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 14 18:10:39.188141 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.188111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:39.195293 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.195268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77vq\" (UniqueName: \"kubernetes.io/projected/f7c8649a-080e-472b-80c7-715d43207da1-kube-api-access-f77vq\") pod \"migrator-7f59b54f6c-nkcpt\" (UID: \"f7c8649a-080e-472b-80c7-715d43207da1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" Jul 14 18:10:39.269888 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.269852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:39.296265 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.296235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77vq\" (UniqueName: \"kubernetes.io/projected/f7c8649a-080e-472b-80c7-715d43207da1-kube-api-access-f77vq\") pod \"migrator-7f59b54f6c-nkcpt\" (UID: \"f7c8649a-080e-472b-80c7-715d43207da1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" Jul 14 18:10:39.309246 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.309219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77vq\" (UniqueName: \"kubernetes.io/projected/f7c8649a-080e-472b-80c7-715d43207da1-kube-api-access-f77vq\") pod \"migrator-7f59b54f6c-nkcpt\" (UID: \"f7c8649a-080e-472b-80c7-715d43207da1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" Jul 14 18:10:39.396039 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.395958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" Jul 14 18:10:39.662032 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.661944 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k7vdr"] Jul 14 18:10:39.677152 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.677119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.679077 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.679046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k7vdr"] Jul 14 18:10:39.680032 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.680009 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 18:10:39.680154 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.680042 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 18:10:39.680154 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.680079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-266xb\"" Jul 14 18:10:39.680154 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.680049 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 18:10:39.681145 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.681125 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 18:10:39.693034 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.693004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:39.699597 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.699568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:39.699758 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.699622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:39.699824 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.699758 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:39.699824 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.699763 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:39.699824 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.699783 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f887d4f-cfwbv: secret "image-registry-tls" not found Jul 14 18:10:39.699824 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.699820 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls podName:8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:43.69980162 +0000 UTC m=+43.131657036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls") pod "dns-default-lg979" (UID: "8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6") : secret "dns-default-metrics-tls" not found Jul 14 18:10:39.700032 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.699839 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls podName:21df3e31-0497-4f60-b03c-edf3cf852737 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:43.699829889 +0000 UTC m=+43.131685304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls") pod "image-registry-658f887d4f-cfwbv" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737") : secret "image-registry-tls" not found Jul 14 18:10:39.775122 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.775089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:39.800223 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.800183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-data-volume\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.800223 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.800215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c797l\" (UniqueName: \"kubernetes.io/projected/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-api-access-c797l\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.800459 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.800242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-crio-socket\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.800459 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.800267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.800459 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.800342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.901146 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.901107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-crio-socket\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.901306 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.901153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.901306 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.901178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.901446 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.901301 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:39.901446 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.901327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-data-volume\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.901446 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.901342 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:39.901446 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.901352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c797l\" (UniqueName: \"kubernetes.io/projected/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-api-access-c797l\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.901446 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.901370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-crio-socket\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.901446 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.901384 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:40.401362436 +0000 UTC m=+39.833217848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:39.901446 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:39.901442 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:40.401414016 +0000 UTC m=+39.833269445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : configmap "kube-rbac-proxy" not found Jul 14 18:10:39.902020 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.901997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-data-volume\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:39.914483 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:39.914413 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c797l\" (UniqueName: \"kubernetes.io/projected/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-api-access-c797l\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:40.115311 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:40.115267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:40.215832 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:40.215640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:40.406330 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:40.406302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:40.406521 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:40.406344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:40.406521 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:40.406470 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:40.406661 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:40.406471 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:40.406661 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:40.406542 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:41.40652246 +0000 UTC m=+40.838377876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : configmap "kube-rbac-proxy" not found Jul 14 18:10:40.406661 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:40.406618 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:41.406603002 +0000 UTC m=+40.838458418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:40.550916 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:40.550886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:40.615828 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:40.615797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:40.821742 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:40.821655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:40.924130 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:40.924094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:41.415399 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:41.415359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:41.415863 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:41.415429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:41.415863 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:41.415518 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:41.415863 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:41.415534 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:41.415863 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:41.415616 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:43.415594744 +0000 UTC m=+42.847450157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:41.415863 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:41.415637 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:43.415629407 +0000 UTC m=+42.847484819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : configmap "kube-rbac-proxy" not found Jul 14 18:10:41.840055 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:41.840031 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84xwb_9c17b1f7-00f3-4d65-941f-16eb0c2e9d24/dns-node-resolver/0.log" Jul 14 18:10:41.888632 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:41.887134 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt"] Jul 14 18:10:41.889843 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:41.889815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:41.891517 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:41.891491 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c8649a_080e_472b_80c7_715d43207da1.slice/crio-564bcd1d3a10700786717757c8fbf0d62814c3f53636feb7571eb7056214c2ef WatchSource:0}: Error finding container 564bcd1d3a10700786717757c8fbf0d62814c3f53636feb7571eb7056214c2ef: Status 404 returned error can't find the container with id 564bcd1d3a10700786717757c8fbf0d62814c3f53636feb7571eb7056214c2ef Jul 14 18:10:41.979246 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:41.979210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:42.409603 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.409485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" event={"ID":"f7c8649a-080e-472b-80c7-715d43207da1","Type":"ContainerStarted","Data":"564bcd1d3a10700786717757c8fbf0d62814c3f53636feb7571eb7056214c2ef"} Jul 14 18:10:42.412017 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.411984 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f03c086-3646-4ade-a3eb-9f7920147f1e" containerID="f263df3b50b8d4e75834e03ea7c577754cf316999157fb5339532acad2d9d43f" exitCode=0 Jul 14 18:10:42.412157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.412053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" event={"ID":"2f03c086-3646-4ade-a3eb-9f7920147f1e","Type":"ContainerDied","Data":"f263df3b50b8d4e75834e03ea7c577754cf316999157fb5339532acad2d9d43f"} Jul 14 18:10:42.413575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.413541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxkkc" event={"ID":"88d206d5-a34a-4da9-8d97-2ccb5c1c707d","Type":"ContainerStarted","Data":"98bb9e1f342e3a3d39bbf757a276482e3fc8c854d7fdd772f34c6ff9f4c3d8a0"} Jul 14 18:10:42.428360 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.428333 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-kxfbj"] Jul 14 18:10:42.445502 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.444881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.448292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.448250 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 14 18:10:42.448459 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.448318 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h7q28\"" Jul 14 18:10:42.448459 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.448344 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 14 18:10:42.448586 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.448510 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 14 18:10:42.448926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.448907 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 14 18:10:42.453186 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.453113 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-kxfbj"] Jul 14 18:10:42.476630 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.476584 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qxkkc" podStartSLOduration=2.211853731 podStartE2EDuration="7.476547603s" totalImagesPullingTime="5.264693872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:42.473118049 +0000 UTC m=+41.904973482" watchObservedRunningTime="2026-07-14 18:10:42.476547603 +0000 UTC m=+41.908403042" Jul 14 18:10:42.479796 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.479768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:42.580012 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.579961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:42.624119 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.624079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8csm\" (UniqueName: \"kubernetes.io/projected/1aec0cdc-bb34-4372-b39b-101082e9fd1b-kube-api-access-d8csm\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.624251 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.624223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1aec0cdc-bb34-4372-b39b-101082e9fd1b-signing-cabundle\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.624305 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.624272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1aec0cdc-bb34-4372-b39b-101082e9fd1b-signing-key\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.725492 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.725455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1aec0cdc-bb34-4372-b39b-101082e9fd1b-signing-cabundle\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.725685 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.725529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1aec0cdc-bb34-4372-b39b-101082e9fd1b-signing-key\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.725685 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.725587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8csm\" (UniqueName: \"kubernetes.io/projected/1aec0cdc-bb34-4372-b39b-101082e9fd1b-kube-api-access-d8csm\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.726241 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.726222 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1aec0cdc-bb34-4372-b39b-101082e9fd1b-signing-cabundle\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.728805 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.728779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1aec0cdc-bb34-4372-b39b-101082e9fd1b-signing-key\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.736969 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.736945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8csm\" (UniqueName: \"kubernetes.io/projected/1aec0cdc-bb34-4372-b39b-101082e9fd1b-kube-api-access-d8csm\") pod \"service-ca-79f68b9f4f-kxfbj\" (UID: \"1aec0cdc-bb34-4372-b39b-101082e9fd1b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.757734 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.757699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" Jul 14 18:10:42.837926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.837903 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cckpq_75b932e3-25bd-4bb8-a092-4de99b1b0625/node-ca/0.log" Jul 14 18:10:42.935456 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:42.935423 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-kxfbj"] Jul 14 18:10:42.940435 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:42.940406 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aec0cdc_bb34_4372_b39b_101082e9fd1b.slice/crio-19a1cf48f33ebccbdc400340e64e672776fab06bf2612a8c1644cb1e9945be72 WatchSource:0}: Error finding container 19a1cf48f33ebccbdc400340e64e672776fab06bf2612a8c1644cb1e9945be72: Status 404 returned error can't find the container with id 19a1cf48f33ebccbdc400340e64e672776fab06bf2612a8c1644cb1e9945be72 Jul 14 18:10:43.095037 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.094945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:43.192723 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.192654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:43.356584 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.356489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:43.420431 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.420394 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f03c086-3646-4ade-a3eb-9f7920147f1e" containerID="84eef0dc4f4c99099f7574da6737418062ea23956e18a39ebe3dbe95657e7a0a" exitCode=0 Jul 14 18:10:43.420629 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.420451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" event={"ID":"2f03c086-3646-4ade-a3eb-9f7920147f1e","Type":"ContainerDied","Data":"84eef0dc4f4c99099f7574da6737418062ea23956e18a39ebe3dbe95657e7a0a"} Jul 14 18:10:43.421841 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.421797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" event={"ID":"1aec0cdc-bb34-4372-b39b-101082e9fd1b","Type":"ContainerStarted","Data":"19a1cf48f33ebccbdc400340e64e672776fab06bf2612a8c1644cb1e9945be72"} Jul 14 18:10:43.431589 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.431536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:43.431989 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.431617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:43.431989 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.431675 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:43.431989 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.431721 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:43.431989 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.431749 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:47.431727657 +0000 UTC m=+46.863583083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:43.431989 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.431766 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:47.431757947 +0000 UTC m=+46.863613361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : configmap "kube-rbac-proxy" not found Jul 14 18:10:43.734111 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.734078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:43.734266 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.734124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:43.734266 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.734242 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:43.734266 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.734261 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f887d4f-cfwbv: secret "image-registry-tls" not found Jul 14 18:10:43.734361 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.734280 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:43.734361 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.734315 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls podName:21df3e31-0497-4f60-b03c-edf3cf852737 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:51.734299742 +0000 UTC m=+51.166155155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls") pod "image-registry-658f887d4f-cfwbv" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737") : secret "image-registry-tls" not found Jul 14 18:10:43.734361 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:43.734328 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls podName:8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:51.734321342 +0000 UTC m=+51.166176754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls") pod "dns-default-lg979" (UID: "8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6") : secret "dns-default-metrics-tls" not found Jul 14 18:10:43.839873 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:43.839842 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxkkc_88d206d5-a34a-4da9-8d97-2ccb5c1c707d/serve-healthcheck-canary/0.log" Jul 14 18:10:44.425867 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:44.425779 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" event={"ID":"f7c8649a-080e-472b-80c7-715d43207da1","Type":"ContainerStarted","Data":"977dad68763bcf5fef94384b5366d790843a0789c275aa44d8e53113ac02f563"} Jul 14 18:10:44.425867 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:44.425855 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" event={"ID":"f7c8649a-080e-472b-80c7-715d43207da1","Type":"ContainerStarted","Data":"8e071b64a6a6b9fc2db27c777f35dd790dd2b8588cab8bd217889eea8a57cf5d"} Jul 14 18:10:44.428920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:44.428890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" event={"ID":"2f03c086-3646-4ade-a3eb-9f7920147f1e","Type":"ContainerStarted","Data":"09df9e54aeab9cebb45d8bab19728abbd09ec009cd6bacf5a5fe7a4c9c20d05c"} Jul 14 18:10:44.444613 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:44.444541 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nkcpt" podStartSLOduration=3.433728093 podStartE2EDuration="5.444525376s" totalImagesPullingTime="2.010797283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:44.444071053 +0000 UTC m=+43.875926489" watchObservedRunningTime="2026-07-14 18:10:44.444525376 +0000 UTC m=+43.876380838" Jul 14 18:10:44.475079 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:44.475021 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bvrtj" podStartSLOduration=6.645054268 podStartE2EDuration="42.47500475s" totalImagesPullingTime="35.829950482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:44.473472462 +0000 UTC m=+43.905327896" watchObservedRunningTime="2026-07-14 18:10:44.47500475 +0000 UTC m=+43.906860184" Jul 14 18:10:46.436444 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:46.436408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" event={"ID":"1aec0cdc-bb34-4372-b39b-101082e9fd1b","Type":"ContainerStarted","Data":"04709c53d05f58f69c837210ae58709b8e7c8886f3de0bb11a12d8e41f138be8"} Jul 14 18:10:46.466292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:46.466239 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-kxfbj" podStartSLOduration=1.534570849 podStartE2EDuration="4.466223546s" totalImagesPullingTime="2.931652697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:46.464575521 +0000 UTC m=+45.896430951" watchObservedRunningTime="2026-07-14 18:10:46.466223546 +0000 UTC m=+45.898078980" Jul 14 18:10:46.791643 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:46.791607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:47.041950 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:47.041833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:47.130137 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:47.130103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:47.216007 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:47.215978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:47.304655 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:47.304570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:47.464517 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:47.464469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:47.465002 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:47.464522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:47.465002 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:47.464660 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:47.465002 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:47.464713 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:47.465002 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:47.464735 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:55.464712171 +0000 UTC m=+54.896567588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:47.465274 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:47.465247 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:55.465226947 +0000 UTC m=+54.897082379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : configmap "kube-rbac-proxy" not found Jul 14 18:10:47.498872 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:47.498787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:47.978836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:47.978803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:48.084419 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:48.084384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:48.476008 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:48.475975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:49.244811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:49.244774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:49.313776 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:49.313738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:49.491177 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:49.491145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:50.283325 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:50.283290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:50.493713 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:50.493676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:51.041050 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.041014 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:51.134028 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.133997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:51.294774 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.294681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:51.680289 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.680206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:51.786530 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.786498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:51.797783 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.797758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:51.797844 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.797800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:51.800106 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.800081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6-metrics-tls\") pod \"dns-default-lg979\" (UID: \"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6\") " pod="openshift-dns/dns-default-lg979" Jul 14 18:10:51.800216 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.800193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"image-registry-658f887d4f-cfwbv\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:51.851961 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.851924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:51.879075 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:51.879035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lg979" Jul 14 18:10:52.005845 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.005783 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-658f887d4f-cfwbv"] Jul 14 18:10:52.008712 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:52.008686 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21df3e31_0497_4f60_b03c_edf3cf852737.slice/crio-aec58c5ef798eb0e465fa97513b71bf22e3170713a551752b6110bbcbd980fea WatchSource:0}: Error finding container aec58c5ef798eb0e465fa97513b71bf22e3170713a551752b6110bbcbd980fea: Status 404 returned error can't find the container with id aec58c5ef798eb0e465fa97513b71bf22e3170713a551752b6110bbcbd980fea Jul 14 18:10:52.027132 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.027106 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lg979"] Jul 14 18:10:52.037537 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:10:52.037515 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee0161c_9fa5_47f2_bbc2_b89afc69f8c6.slice/crio-d6b5b95b4c1ede16506073a2743ac573e58049ca7c10891e895e37bab0b998e0 WatchSource:0}: Error finding container d6b5b95b4c1ede16506073a2743ac573e58049ca7c10891e895e37bab0b998e0: Status 404 returned error can't find the container with id d6b5b95b4c1ede16506073a2743ac573e58049ca7c10891e895e37bab0b998e0 Jul 14 18:10:52.163836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.163793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:52.289174 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.289143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:52.407318 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.407284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:52.456900 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.456854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" event={"ID":"21df3e31-0497-4f60-b03c-edf3cf852737","Type":"ContainerStarted","Data":"dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b"} Jul 14 18:10:52.457135 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.456924 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" event={"ID":"21df3e31-0497-4f60-b03c-edf3cf852737","Type":"ContainerStarted","Data":"aec58c5ef798eb0e465fa97513b71bf22e3170713a551752b6110bbcbd980fea"} Jul 14 18:10:52.457135 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.456994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:10:52.458097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.458073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lg979" event={"ID":"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6","Type":"ContainerStarted","Data":"d6b5b95b4c1ede16506073a2743ac573e58049ca7c10891e895e37bab0b998e0"} Jul 14 18:10:52.493351 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.493303 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" podStartSLOduration=45.493285122 podStartE2EDuration="45.493285122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:52.491609908 +0000 UTC m=+51.923465343" watchObservedRunningTime="2026-07-14 18:10:52.493285122 +0000 UTC m=+51.925140555" Jul 14 18:10:52.498780 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.498757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:52.683266 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.683185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:52.909104 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:52.909057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:53.087277 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:53.087243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:53.278883 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:53.278832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:53.692482 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:53.692442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:54.291523 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:54.291433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:54.466707 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:54.466667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lg979" event={"ID":"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6","Type":"ContainerStarted","Data":"1e1a5ff69778947b480b5b44ab70a954298d3d303f39cbcda7c5d184b9798c33"} Jul 14 18:10:54.466949 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:54.466751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lg979" event={"ID":"8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6","Type":"ContainerStarted","Data":"cb453179f18b214c34613910c0eac3bc4cb43fe0ed2c8bd89b627d696fbf3b59"} Jul 14 18:10:54.466949 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:54.466830 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lg979" Jul 14 18:10:54.518837 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:54.518782 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lg979" podStartSLOduration=17.724411263 podStartE2EDuration="19.518764136s" totalImagesPullingTime="1.794352873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:54.517974206 +0000 UTC m=+53.949829640" watchObservedRunningTime="2026-07-14 18:10:54.518764136 +0000 UTC m=+53.950619624" Jul 14 18:10:54.552128 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:54.552048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:54.689267 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:54.689231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:54.876474 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:54.876385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:55.501079 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:55.501042 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:55.530523 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:55.530496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:55.530704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:55.530537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:55.530704 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:55.530676 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:55.530816 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:10:55.530740 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls podName:7ee50913-fbec-42f0-a1e8-55242e9c5ff7 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:11.530718887 +0000 UTC m=+70.962574299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k7vdr" (UID: "7ee50913-fbec-42f0-a1e8-55242e9c5ff7") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:55.531122 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:55.531104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:10:55.884130 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:55.884045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:56.280821 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:56.280787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:56.427331 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:56.427294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:56.526921 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:56.526884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:56.623645 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:56.623536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:56.972673 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:56.972643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:57.072670 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:57.072640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:57.176362 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:57.176328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:57.276782 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:57.276698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:57.477440 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:57.477402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:57.581170 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:57.581086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:57.916055 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:57.915970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:58.016786 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:58.016755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:58.104815 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:10:58.104782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:01.384591 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:01.384563 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5gxk" Jul 14 18:11:03.242172 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.242127 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz"] Jul 14 18:11:03.270397 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.270363 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9"] Jul 14 18:11:03.270594 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.270521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" Jul 14 18:11:03.273858 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.273782 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 18:11:03.273858 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.273788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 18:11:03.274907 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.274889 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 14 18:11:03.275164 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.275152 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kkdz6\"" Jul 14 18:11:03.275756 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.275737 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 18:11:03.285979 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.285958 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-658f887d4f-cfwbv"] Jul 14 18:11:03.286103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.285994 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz"] Jul 14 18:11:03.286103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.286064 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.286209 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.286102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9"] Jul 14 18:11:03.286209 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.286139 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65f97494db-wtzsp"] Jul 14 18:11:03.291536 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.291471 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 14 18:11:03.306818 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.306790 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f97494db-wtzsp"] Jul 14 18:11:03.306954 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.306878 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.343105 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.343078 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-wfqbp"] Jul 14 18:11:03.367528 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.367503 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-wfqbp"] Jul 14 18:11:03.367697 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.367619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-wfqbp" Jul 14 18:11:03.370733 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.370709 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 18:11:03.370879 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.370803 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-qt8dm\"" Jul 14 18:11:03.370879 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.370809 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 18:11:03.394858 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.394831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-tmp\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.395006 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.394862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjjl\" (UniqueName: \"kubernetes.io/projected/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-kube-api-access-8fjjl\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.395006 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.394881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-registry-tls\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.395006 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.394899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e316ae5-cb89-435b-90d9-1e366152d608-registry-certificates\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.395006 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.394991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74nkw\" (UniqueName: \"kubernetes.io/projected/97ea8a28-bb82-4f10-979e-468b9258505b-kube-api-access-74nkw\") pod \"managed-serviceaccount-addon-agent-776d776b96-r2cmz\" (UID: \"97ea8a28-bb82-4f10-979e-468b9258505b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" Jul 14 18:11:03.395157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.395033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh88g\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-kube-api-access-rh88g\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.395157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.395062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-klusterlet-config\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.395157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.395093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e316ae5-cb89-435b-90d9-1e366152d608-trusted-ca\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.395157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.395111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97ea8a28-bb82-4f10-979e-468b9258505b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776d776b96-r2cmz\" (UID: \"97ea8a28-bb82-4f10-979e-468b9258505b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" Jul 14 18:11:03.395157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.395147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-bound-sa-token\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.395332 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.395182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e316ae5-cb89-435b-90d9-1e366152d608-image-registry-private-configuration\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.395332 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.395214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e316ae5-cb89-435b-90d9-1e366152d608-ca-trust-extracted\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.395332 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.395230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e316ae5-cb89-435b-90d9-1e366152d608-installation-pull-secrets\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.496327 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e316ae5-cb89-435b-90d9-1e366152d608-installation-pull-secrets\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.496327 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-tmp\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.496575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjjl\" (UniqueName: \"kubernetes.io/projected/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-kube-api-access-8fjjl\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.496575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-registry-tls\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.496575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e316ae5-cb89-435b-90d9-1e366152d608-registry-certificates\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.496575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74nkw\" (UniqueName: \"kubernetes.io/projected/97ea8a28-bb82-4f10-979e-468b9258505b-kube-api-access-74nkw\") pod \"managed-serviceaccount-addon-agent-776d776b96-r2cmz\" (UID: \"97ea8a28-bb82-4f10-979e-468b9258505b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" Jul 14 18:11:03.496575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh88g\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-kube-api-access-rh88g\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.496575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-klusterlet-config\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.496876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e316ae5-cb89-435b-90d9-1e366152d608-trusted-ca\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.496876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97ea8a28-bb82-4f10-979e-468b9258505b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776d776b96-r2cmz\" (UID: \"97ea8a28-bb82-4f10-979e-468b9258505b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" Jul 14 18:11:03.496876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-bound-sa-token\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.496876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cb5\" (UniqueName: \"kubernetes.io/projected/e9024dcf-185d-412a-8d84-42605eca593a-kube-api-access-p2cb5\") pod \"downloads-68c9cbb497-wfqbp\" (UID: \"e9024dcf-185d-412a-8d84-42605eca593a\") " pod="openshift-console/downloads-68c9cbb497-wfqbp" Jul 14 18:11:03.496876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e316ae5-cb89-435b-90d9-1e366152d608-image-registry-private-configuration\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.496876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-tmp\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.496876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.496827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e316ae5-cb89-435b-90d9-1e366152d608-ca-trust-extracted\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.497635 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.497612 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e316ae5-cb89-435b-90d9-1e366152d608-registry-certificates\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.497866 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.497846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e316ae5-cb89-435b-90d9-1e366152d608-ca-trust-extracted\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.498057 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.498035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e316ae5-cb89-435b-90d9-1e366152d608-trusted-ca\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.499220 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.499194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e316ae5-cb89-435b-90d9-1e366152d608-image-registry-private-configuration\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.499423 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.499404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e316ae5-cb89-435b-90d9-1e366152d608-installation-pull-secrets\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.499496 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.499480 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-registry-tls\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.499644 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.499624 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97ea8a28-bb82-4f10-979e-468b9258505b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776d776b96-r2cmz\" (UID: \"97ea8a28-bb82-4f10-979e-468b9258505b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" Jul 14 18:11:03.499691 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.499674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-klusterlet-config\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.511917 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.511883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh88g\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-kube-api-access-rh88g\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.511917 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.511908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjjl\" (UniqueName: \"kubernetes.io/projected/0b396aae-c1a4-46b7-bd0c-599d242e4b5d-kube-api-access-8fjjl\") pod \"klusterlet-addon-workmgr-7656fd6cf5-2cbs9\" (UID: \"0b396aae-c1a4-46b7-bd0c-599d242e4b5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.512467 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.512439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e316ae5-cb89-435b-90d9-1e366152d608-bound-sa-token\") pod \"image-registry-65f97494db-wtzsp\" (UID: \"4e316ae5-cb89-435b-90d9-1e366152d608\") " pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.513546 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.513525 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74nkw\" (UniqueName: \"kubernetes.io/projected/97ea8a28-bb82-4f10-979e-468b9258505b-kube-api-access-74nkw\") pod \"managed-serviceaccount-addon-agent-776d776b96-r2cmz\" (UID: \"97ea8a28-bb82-4f10-979e-468b9258505b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" Jul 14 18:11:03.590967 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.590931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" Jul 14 18:11:03.597675 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.597638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cb5\" (UniqueName: \"kubernetes.io/projected/e9024dcf-185d-412a-8d84-42605eca593a-kube-api-access-p2cb5\") pod \"downloads-68c9cbb497-wfqbp\" (UID: \"e9024dcf-185d-412a-8d84-42605eca593a\") " pod="openshift-console/downloads-68c9cbb497-wfqbp" Jul 14 18:11:03.598024 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.597995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:03.608372 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.608353 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cb5\" (UniqueName: \"kubernetes.io/projected/e9024dcf-185d-412a-8d84-42605eca593a-kube-api-access-p2cb5\") pod \"downloads-68c9cbb497-wfqbp\" (UID: \"e9024dcf-185d-412a-8d84-42605eca593a\") " pod="openshift-console/downloads-68c9cbb497-wfqbp" Jul 14 18:11:03.615170 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.615139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:03.677174 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.676865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-wfqbp" Jul 14 18:11:03.749043 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.747779 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-678b44b47f-gl7cw"] Jul 14 18:11:03.751714 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.751666 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.755129 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.754935 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 18:11:03.758977 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.758841 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 18:11:03.758977 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.758877 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z5ljq\"" Jul 14 18:11:03.760368 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.760282 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz"] Jul 14 18:11:03.763501 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.763463 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 18:11:03.764179 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.764035 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 18:11:03.764379 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.764324 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 18:11:03.764535 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:03.764509 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97ea8a28_bb82_4f10_979e_468b9258505b.slice/crio-d9af35b6a770a79eaa8ab6c690ad7b61994225d072e66154ddea3e36e612c2d2 WatchSource:0}: Error finding container d9af35b6a770a79eaa8ab6c690ad7b61994225d072e66154ddea3e36e612c2d2: Status 404 returned error can't find the container with id d9af35b6a770a79eaa8ab6c690ad7b61994225d072e66154ddea3e36e612c2d2 Jul 14 18:11:03.775991 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.775718 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678b44b47f-gl7cw"] Jul 14 18:11:03.791691 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.791661 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9"] Jul 14 18:11:03.795515 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:03.795480 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b396aae_c1a4_46b7_bd0c_599d242e4b5d.slice/crio-6b8aa06f1ff9ad9e6b92167196e72308e9b96a6dabe1a8d5e738244035838413 WatchSource:0}: Error finding container 6b8aa06f1ff9ad9e6b92167196e72308e9b96a6dabe1a8d5e738244035838413: Status 404 returned error can't find the container with id 6b8aa06f1ff9ad9e6b92167196e72308e9b96a6dabe1a8d5e738244035838413 Jul 14 18:11:03.800283 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.800254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-service-ca\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.800376 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.800329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tzx\" (UniqueName: \"kubernetes.io/projected/978f8353-4ae5-4b5b-a46a-c52d51055c63-kube-api-access-h5tzx\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.800442 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.800426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-serving-cert\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.800484 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.800458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-oauth-config\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.800525 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.800495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-oauth-serving-cert\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.800607 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.800580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-config\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.824113 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.824085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f97494db-wtzsp"] Jul 14 18:11:03.827392 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:03.827368 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e316ae5_cb89_435b_90d9_1e366152d608.slice/crio-1558affd3afb5f44903d4be8a3f96b190ea3300d7a69233c07ed55de4ce6bcf1 WatchSource:0}: Error finding container 1558affd3afb5f44903d4be8a3f96b190ea3300d7a69233c07ed55de4ce6bcf1: Status 404 returned error can't find the container with id 1558affd3afb5f44903d4be8a3f96b190ea3300d7a69233c07ed55de4ce6bcf1 Jul 14 18:11:03.837135 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.837046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-wfqbp"] Jul 14 18:11:03.839781 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:03.839761 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9024dcf_185d_412a_8d84_42605eca593a.slice/crio-84e78e1eec155498aa5b50b39abbd49f66a559615c8c0c5d62585bf5942571ec WatchSource:0}: Error finding container 84e78e1eec155498aa5b50b39abbd49f66a559615c8c0c5d62585bf5942571ec: Status 404 returned error can't find the container with id 84e78e1eec155498aa5b50b39abbd49f66a559615c8c0c5d62585bf5942571ec Jul 14 18:11:03.900981 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.900953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-serving-cert\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.900981 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.900984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-oauth-config\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.901426 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.901011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-oauth-serving-cert\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.901426 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.901047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-config\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.901426 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.901074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-service-ca\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.901426 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.901110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tzx\" (UniqueName: \"kubernetes.io/projected/978f8353-4ae5-4b5b-a46a-c52d51055c63-kube-api-access-h5tzx\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.901856 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.901834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-config\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.901936 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.901859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-service-ca\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.901975 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.901950 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-oauth-serving-cert\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.903853 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.903830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-serving-cert\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.903931 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.903874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-oauth-config\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:03.915885 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:03.915860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tzx\" (UniqueName: \"kubernetes.io/projected/978f8353-4ae5-4b5b-a46a-c52d51055c63-kube-api-access-h5tzx\") pod \"console-678b44b47f-gl7cw\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:04.066968 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.066885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:04.238368 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.238344 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678b44b47f-gl7cw"] Jul 14 18:11:04.240181 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:04.240146 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978f8353_4ae5_4b5b_a46a_c52d51055c63.slice/crio-4994d9a02645295dd544388a4a7225b54f4d4a159d3ef5a563930a0262ddd058 WatchSource:0}: Error finding container 4994d9a02645295dd544388a4a7225b54f4d4a159d3ef5a563930a0262ddd058: Status 404 returned error can't find the container with id 4994d9a02645295dd544388a4a7225b54f4d4a159d3ef5a563930a0262ddd058 Jul 14 18:11:04.472290 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.472265 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lg979" Jul 14 18:11:04.496665 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.496632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" event={"ID":"0b396aae-c1a4-46b7-bd0c-599d242e4b5d","Type":"ContainerStarted","Data":"6b8aa06f1ff9ad9e6b92167196e72308e9b96a6dabe1a8d5e738244035838413"} Jul 14 18:11:04.497800 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.497772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" event={"ID":"97ea8a28-bb82-4f10-979e-468b9258505b","Type":"ContainerStarted","Data":"d9af35b6a770a79eaa8ab6c690ad7b61994225d072e66154ddea3e36e612c2d2"} Jul 14 18:11:04.498928 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.498903 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678b44b47f-gl7cw" event={"ID":"978f8353-4ae5-4b5b-a46a-c52d51055c63","Type":"ContainerStarted","Data":"4994d9a02645295dd544388a4a7225b54f4d4a159d3ef5a563930a0262ddd058"} Jul 14 18:11:04.499998 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.499974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-wfqbp" event={"ID":"e9024dcf-185d-412a-8d84-42605eca593a","Type":"ContainerStarted","Data":"84e78e1eec155498aa5b50b39abbd49f66a559615c8c0c5d62585bf5942571ec"} Jul 14 18:11:04.501355 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.501334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f97494db-wtzsp" event={"ID":"4e316ae5-cb89-435b-90d9-1e366152d608","Type":"ContainerStarted","Data":"44fd1705dd2cc3af2ee6e4b2188d7a8dd7c4a63a9d1f081a565de13bffaf885c"} Jul 14 18:11:04.501429 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.501372 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f97494db-wtzsp" event={"ID":"4e316ae5-cb89-435b-90d9-1e366152d608","Type":"ContainerStarted","Data":"1558affd3afb5f44903d4be8a3f96b190ea3300d7a69233c07ed55de4ce6bcf1"} Jul 14 18:11:04.501508 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.501491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:04.536192 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:04.536135 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65f97494db-wtzsp" podStartSLOduration=1.536117923 podStartE2EDuration="1.536117923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:04.535955014 +0000 UTC m=+63.967810448" watchObservedRunningTime="2026-07-14 18:11:04.536117923 +0000 UTC m=+63.967973357" Jul 14 18:11:06.415938 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.415869 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr"] Jul 14 18:11:06.425224 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.425197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" Jul 14 18:11:06.429119 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.428752 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z4d8z\"" Jul 14 18:11:06.429119 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.428973 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 14 18:11:06.437527 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.437257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr"] Jul 14 18:11:06.526529 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.526491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/78661dc5-00ea-49ca-866c-4e82cb3eead9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-flnzr\" (UID: \"78661dc5-00ea-49ca-866c-4e82cb3eead9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" Jul 14 18:11:06.628089 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.627515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/78661dc5-00ea-49ca-866c-4e82cb3eead9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-flnzr\" (UID: \"78661dc5-00ea-49ca-866c-4e82cb3eead9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" Jul 14 18:11:06.628089 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:11:06.627748 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 14 18:11:06.628089 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:11:06.627844 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78661dc5-00ea-49ca-866c-4e82cb3eead9-tls-certificates podName:78661dc5-00ea-49ca-866c-4e82cb3eead9 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:07.127799987 +0000 UTC m=+66.559655401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/78661dc5-00ea-49ca-866c-4e82cb3eead9-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-flnzr" (UID: "78661dc5-00ea-49ca-866c-4e82cb3eead9") : secret "prometheus-operator-admission-webhook-tls" not found Jul 14 18:11:06.830267 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.830147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:11:06.833351 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.833144 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:11:06.843546 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.843484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067d5347-e149-498f-b815-f682137d2b41-metrics-certs\") pod \"network-metrics-daemon-t9k4k\" (UID: \"067d5347-e149-498f-b815-f682137d2b41\") " pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:11:06.931637 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.931205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:11:06.931637 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.931299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:11:06.934700 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.934672 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:11:06.934955 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.934940 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:11:06.945117 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.944913 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:11:06.947174 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.947109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78631d62-ebdf-43af-a039-83ba0683e841-original-pull-secret\") pod \"global-pull-secret-syncer-82p7g\" (UID: \"78631d62-ebdf-43af-a039-83ba0683e841\") " pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:11:06.958502 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:06.958445 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7fh\" (UniqueName: \"kubernetes.io/projected/74c6b78c-4275-40e0-aa17-e14f72c95cce-kube-api-access-vb7fh\") pod \"network-check-target-wjkg4\" (UID: \"74c6b78c-4275-40e0-aa17-e14f72c95cce\") " pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:11:07.115053 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:07.114971 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jkhfz\"" Jul 14 18:11:07.117320 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:07.117296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82p7g" Jul 14 18:11:07.122677 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:07.122400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t9k4k" Jul 14 18:11:07.125844 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:07.125570 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5vgxh\"" Jul 14 18:11:07.133042 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:07.133019 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:11:07.133628 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:07.133337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/78661dc5-00ea-49ca-866c-4e82cb3eead9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-flnzr\" (UID: \"78661dc5-00ea-49ca-866c-4e82cb3eead9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" Jul 14 18:11:07.139272 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:07.139223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/78661dc5-00ea-49ca-866c-4e82cb3eead9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-flnzr\" (UID: \"78661dc5-00ea-49ca-866c-4e82cb3eead9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" Jul 14 18:11:07.344141 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:07.344105 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" Jul 14 18:11:10.191500 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.190992 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr"] Jul 14 18:11:10.215436 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.215402 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82p7g"] Jul 14 18:11:10.219108 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:10.219064 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78631d62_ebdf_43af_a039_83ba0683e841.slice/crio-d92531fdd9f6199e3f147de0d33ba46e94198a23174502ea2d54ec65ef574cd7 WatchSource:0}: Error finding container d92531fdd9f6199e3f147de0d33ba46e94198a23174502ea2d54ec65ef574cd7: Status 404 returned error can't find the container with id d92531fdd9f6199e3f147de0d33ba46e94198a23174502ea2d54ec65ef574cd7 Jul 14 18:11:10.250847 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.250805 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wjkg4"] Jul 14 18:11:10.254750 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:10.254511 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c6b78c_4275_40e0_aa17_e14f72c95cce.slice/crio-b2e6540a598dc7feaf953cc0551940b3045104d9ffa20d9d10b1d4c6283e9b04 WatchSource:0}: Error finding container b2e6540a598dc7feaf953cc0551940b3045104d9ffa20d9d10b1d4c6283e9b04: Status 404 returned error can't find the container with id b2e6540a598dc7feaf953cc0551940b3045104d9ffa20d9d10b1d4c6283e9b04 Jul 14 18:11:10.408243 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.408222 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t9k4k"] Jul 14 18:11:10.410363 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:10.410340 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067d5347_e149_498f_b815_f682137d2b41.slice/crio-1e6cc1e7f2fc4fbb3e8c75fc276c5166098e3e8346c5a9326c7b578bca546a25 WatchSource:0}: Error finding container 1e6cc1e7f2fc4fbb3e8c75fc276c5166098e3e8346c5a9326c7b578bca546a25: Status 404 returned error can't find the container with id 1e6cc1e7f2fc4fbb3e8c75fc276c5166098e3e8346c5a9326c7b578bca546a25 Jul 14 18:11:10.533337 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.533299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" event={"ID":"0b396aae-c1a4-46b7-bd0c-599d242e4b5d","Type":"ContainerStarted","Data":"dbec48cd860aeade204ffcbf49949bd4c234e4f79c0901a46d3793566bc541e2"} Jul 14 18:11:10.533537 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.533470 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:10.535606 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.535583 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" Jul 14 18:11:10.535917 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.535872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" event={"ID":"97ea8a28-bb82-4f10-979e-468b9258505b","Type":"ContainerStarted","Data":"65815ac1818770cf84979189e277ce78afbf0b8ed53915b26da13057c265db48"} Jul 14 18:11:10.538286 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.537487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82p7g" event={"ID":"78631d62-ebdf-43af-a039-83ba0683e841","Type":"ContainerStarted","Data":"d92531fdd9f6199e3f147de0d33ba46e94198a23174502ea2d54ec65ef574cd7"} Jul 14 18:11:10.539569 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.539529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t9k4k" event={"ID":"067d5347-e149-498f-b815-f682137d2b41","Type":"ContainerStarted","Data":"1e6cc1e7f2fc4fbb3e8c75fc276c5166098e3e8346c5a9326c7b578bca546a25"} Jul 14 18:11:10.541281 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.541246 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wjkg4" event={"ID":"74c6b78c-4275-40e0-aa17-e14f72c95cce","Type":"ContainerStarted","Data":"b2e6540a598dc7feaf953cc0551940b3045104d9ffa20d9d10b1d4c6283e9b04"} Jul 14 18:11:10.542535 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.542512 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" event={"ID":"78661dc5-00ea-49ca-866c-4e82cb3eead9","Type":"ContainerStarted","Data":"9fe68b6b287d02617afc5e48fb59888e865a16d4eae22caafd145f4f0c4c9198"} Jul 14 18:11:10.544307 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.544264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678b44b47f-gl7cw" event={"ID":"978f8353-4ae5-4b5b-a46a-c52d51055c63","Type":"ContainerStarted","Data":"3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9"} Jul 14 18:11:10.558161 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.558110 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7656fd6cf5-2cbs9" podStartSLOduration=1.37197241 podStartE2EDuration="7.55809442s" totalImagesPullingTime="6.18612201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:10.556905826 +0000 UTC m=+69.988761263" watchObservedRunningTime="2026-07-14 18:11:10.55809442 +0000 UTC m=+69.989949854" Jul 14 18:11:10.578092 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.578033 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776d776b96-r2cmz" podStartSLOduration=1.382341142 podStartE2EDuration="7.578014827s" totalImagesPullingTime="6.195673685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:10.577930862 +0000 UTC m=+70.009786298" watchObservedRunningTime="2026-07-14 18:11:10.578014827 +0000 UTC m=+70.009870255" Jul 14 18:11:10.655103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:10.655046 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-678b44b47f-gl7cw" podStartSLOduration=1.924089744 podStartE2EDuration="7.655024475s" totalImagesPullingTime="5.730934731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:10.653883362 +0000 UTC m=+70.085738797" watchObservedRunningTime="2026-07-14 18:11:10.655024475 +0000 UTC m=+70.086879908" Jul 14 18:11:11.586044 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:11.585479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:11:11.593815 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:11.593748 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7ee50913-fbec-42f0-a1e8-55242e9c5ff7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7vdr\" (UID: \"7ee50913-fbec-42f0-a1e8-55242e9c5ff7\") " pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:11:11.798600 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:11.798345 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-266xb\"" Jul 14 18:11:11.807046 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:11.806789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k7vdr" Jul 14 18:11:12.244071 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.243948 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k7vdr"] Jul 14 18:11:12.249490 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:12.249444 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee50913_fbec_42f0_a1e8_55242e9c5ff7.slice/crio-b6593eea70c9403b18cc8a0fe889913b57f0756be48b9b079918f08f1f364aa6 WatchSource:0}: Error finding container b6593eea70c9403b18cc8a0fe889913b57f0756be48b9b079918f08f1f364aa6: Status 404 returned error can't find the container with id b6593eea70c9403b18cc8a0fe889913b57f0756be48b9b079918f08f1f364aa6 Jul 14 18:11:12.559079 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.559038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t9k4k" event={"ID":"067d5347-e149-498f-b815-f682137d2b41","Type":"ContainerStarted","Data":"cbb0d184a45c7efb05ad348dd2aa72b45997b170d46dd4ededb440ffb89ef55f"} Jul 14 18:11:12.559256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.559108 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t9k4k" event={"ID":"067d5347-e149-498f-b815-f682137d2b41","Type":"ContainerStarted","Data":"5063f8f4d83af08eec138a0e2f1a20d620acc53291f23ddcdce9b4ae78efe3df"} Jul 14 18:11:12.561070 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.561037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" event={"ID":"78661dc5-00ea-49ca-866c-4e82cb3eead9","Type":"ContainerStarted","Data":"2e72db199cf29dba7ba55c8cf8b6775f1e1c025ff58e5ebe53338b358657b614"} Jul 14 18:11:12.561414 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.561396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" Jul 14 18:11:12.563275 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.562931 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k7vdr" event={"ID":"7ee50913-fbec-42f0-a1e8-55242e9c5ff7","Type":"ContainerStarted","Data":"b6593eea70c9403b18cc8a0fe889913b57f0756be48b9b079918f08f1f364aa6"} Jul 14 18:11:12.567716 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.567627 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" Jul 14 18:11:12.590672 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.590520 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t9k4k" podStartSLOduration=68.937892992 podStartE2EDuration="1m10.590503835s" totalImagesPullingTime="1.652610843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:12.588249223 +0000 UTC m=+72.020104658" watchObservedRunningTime="2026-07-14 18:11:12.590503835 +0000 UTC m=+72.022359270" Jul 14 18:11:12.610572 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:12.610498 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-flnzr" podStartSLOduration=4.747763977 podStartE2EDuration="6.610458978s" totalImagesPullingTime="1.862695001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:12.60831512 +0000 UTC m=+72.040170558" watchObservedRunningTime="2026-07-14 18:11:12.610458978 +0000 UTC m=+72.042314412" Jul 14 18:11:13.296330 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:13.296084 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:11:14.067998 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:14.067752 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:14.067998 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:14.067814 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:14.075217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:14.075174 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:14.581301 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:14.581271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:11:19.602141 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:19.602047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k7vdr" event={"ID":"7ee50913-fbec-42f0-a1e8-55242e9c5ff7","Type":"ContainerStarted","Data":"60d864da53b64aeb212fc8c37082df0dd1087c715c386a58ec805fce32c7f33d"} Jul 14 18:11:19.603739 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:19.603708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82p7g" event={"ID":"78631d62-ebdf-43af-a039-83ba0683e841","Type":"ContainerStarted","Data":"d1e45461237644f56e774c7c5a45a894462f5a196a4f77d28302c4ad38b58025"} Jul 14 18:11:19.605281 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:19.605253 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wjkg4" event={"ID":"74c6b78c-4275-40e0-aa17-e14f72c95cce","Type":"ContainerStarted","Data":"b62fb6453cbeb2c27dff12f5c3868fcfcd4b0ebc14603e85f14101c41affa814"} Jul 14 18:11:19.605420 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:19.605387 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:11:19.625751 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:19.625687 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-82p7g" podStartSLOduration=68.536780206 podStartE2EDuration="1m17.625670836s" totalImagesPullingTime="9.08889063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:19.625108759 +0000 UTC m=+79.056964195" watchObservedRunningTime="2026-07-14 18:11:19.625670836 +0000 UTC m=+79.057526270" Jul 14 18:11:19.649292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:19.649241 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wjkg4" podStartSLOduration=68.596728265 podStartE2EDuration="1m17.649221319s" totalImagesPullingTime="9.052493054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:19.64723049 +0000 UTC m=+79.079085924" watchObservedRunningTime="2026-07-14 18:11:19.649221319 +0000 UTC m=+79.081076752" Jul 14 18:11:25.513714 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.513173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65f97494db-wtzsp" Jul 14 18:11:25.867673 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.867580 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kdfnk"] Jul 14 18:11:25.934243 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.934207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:25.939579 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.939413 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 18:11:25.939885 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.939864 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 18:11:25.940381 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.940167 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 18:11:25.940381 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.940187 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 18:11:25.940542 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.940521 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 18:11:25.940801 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.940782 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 18:11:25.941429 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:25.941250 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pj49j\"" Jul 14 18:11:26.009803 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.009765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.009986 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.009832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-wtmp\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.009986 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.009857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-root\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.009986 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.009879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-textfile\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.009986 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.009911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7xg\" (UniqueName: \"kubernetes.io/projected/48386c71-621c-42dd-b9f9-5be94e9f50ad-kube-api-access-md7xg\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.009986 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.009934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-tls\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.009986 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.009962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.010315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.010003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-sys\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.010315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.010030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48386c71-621c-42dd-b9f9-5be94e9f50ad-metrics-client-ca\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.110872 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.110835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-sys\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.110872 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.110885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48386c71-621c-42dd-b9f9-5be94e9f50ad-metrics-client-ca\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.110920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.110965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-sys\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.110970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-wtmp\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-wtmp\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-root\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111362 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-textfile\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111362 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48386c71-621c-42dd-b9f9-5be94e9f50ad-root\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111362 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7xg\" (UniqueName: \"kubernetes.io/projected/48386c71-621c-42dd-b9f9-5be94e9f50ad-kube-api-access-md7xg\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111362 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-tls\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111362 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111622 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-textfile\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111696 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111872 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.111692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48386c71-621c-42dd-b9f9-5be94e9f50ad-metrics-client-ca\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.111872 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:11:26.111777 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 18:11:26.111872 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:11:26.111842 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-tls podName:48386c71-621c-42dd-b9f9-5be94e9f50ad nodeName:}" failed. No retries permitted until 2026-07-14 18:11:26.611823096 +0000 UTC m=+86.043678514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-tls") pod "node-exporter-kdfnk" (UID: "48386c71-621c-42dd-b9f9-5be94e9f50ad") : secret "node-exporter-tls" not found Jul 14 18:11:26.113823 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.113797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.127047 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.126966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7xg\" (UniqueName: \"kubernetes.io/projected/48386c71-621c-42dd-b9f9-5be94e9f50ad-kube-api-access-md7xg\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.616860 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.616828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-tls\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.619448 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.619418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48386c71-621c-42dd-b9f9-5be94e9f50ad-node-exporter-tls\") pod \"node-exporter-kdfnk\" (UID: \"48386c71-621c-42dd-b9f9-5be94e9f50ad\") " pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.845657 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:26.845619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kdfnk" Jul 14 18:11:26.950181 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:11:26.950136 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48386c71_621c_42dd_b9f9_5be94e9f50ad.slice/crio-a5f1fd204855de5baf2b4c1d8ff7248303c434d8cfa164490f38c95de722b08c WatchSource:0}: Error finding container a5f1fd204855de5baf2b4c1d8ff7248303c434d8cfa164490f38c95de722b08c: Status 404 returned error can't find the container with id a5f1fd204855de5baf2b4c1d8ff7248303c434d8cfa164490f38c95de722b08c Jul 14 18:11:27.638192 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:27.638137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdfnk" event={"ID":"48386c71-621c-42dd-b9f9-5be94e9f50ad","Type":"ContainerStarted","Data":"a5f1fd204855de5baf2b4c1d8ff7248303c434d8cfa164490f38c95de722b08c"} Jul 14 18:11:27.639918 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:27.639883 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-wfqbp" event={"ID":"e9024dcf-185d-412a-8d84-42605eca593a","Type":"ContainerStarted","Data":"6eb317acf9a21ac113d8fd30dc9ded396f61449dca8e5131492ce99d89b20b3a"} Jul 14 18:11:28.306460 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.306419 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" podUID="21df3e31-0497-4f60-b03c-edf3cf852737" containerName="registry" containerID="cri-o://dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b" gracePeriod=30 Jul 14 18:11:28.632996 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.632961 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:11:28.645994 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.645957 2579 generic.go:356] "Generic (PLEG): container finished" podID="21df3e31-0497-4f60-b03c-edf3cf852737" containerID="dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b" exitCode=0 Jul 14 18:11:28.646431 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.646007 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" Jul 14 18:11:28.646431 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.646050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" event={"ID":"21df3e31-0497-4f60-b03c-edf3cf852737","Type":"ContainerDied","Data":"dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b"} Jul 14 18:11:28.646431 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.646118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-658f887d4f-cfwbv" event={"ID":"21df3e31-0497-4f60-b03c-edf3cf852737","Type":"ContainerDied","Data":"aec58c5ef798eb0e465fa97513b71bf22e3170713a551752b6110bbcbd980fea"} Jul 14 18:11:28.646431 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.646168 2579 scope.go:117] "RemoveContainer" containerID="dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b" Jul 14 18:11:28.648769 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.648747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k7vdr" event={"ID":"7ee50913-fbec-42f0-a1e8-55242e9c5ff7","Type":"ContainerStarted","Data":"892afafb7b2cd8acccf75a1713de6057fbc0851e1c2ecc1de923ec1dbe5a146a"} Jul 14 18:11:28.650891 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.650867 2579 generic.go:356] "Generic (PLEG): container finished" podID="48386c71-621c-42dd-b9f9-5be94e9f50ad" containerID="a44eb6dbabe836462a1f95262452b1dca31a217c5ee314976f225bb3428d97f4" exitCode=0 Jul 14 18:11:28.651008 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.650964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdfnk" event={"ID":"48386c71-621c-42dd-b9f9-5be94e9f50ad","Type":"ContainerDied","Data":"a44eb6dbabe836462a1f95262452b1dca31a217c5ee314976f225bb3428d97f4"} Jul 14 18:11:28.651585 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.651526 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-wfqbp" Jul 14 18:11:28.656773 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.656753 2579 scope.go:117] "RemoveContainer" containerID="dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b" Jul 14 18:11:28.657144 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:11:28.657117 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b\": container with ID starting with dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b not found: ID does not exist" containerID="dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b" Jul 14 18:11:28.657238 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.657154 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b"} err="failed to get container status \"dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b\": rpc error: code = NotFound desc = could not find container \"dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b\": container with ID starting with dc133acba6ea334496b765c65d631edbdb96ca4a5745d8425246447f5fdf8b7b not found: ID does not exist" Jul 14 18:11:28.664208 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.664184 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-wfqbp" Jul 14 18:11:28.729026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.728975 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-wfqbp" podStartSLOduration=2.219854275 podStartE2EDuration="25.728955157s" totalImagesPullingTime="23.509100882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:28.727424732 +0000 UTC m=+88.159280180" watchObservedRunningTime="2026-07-14 18:11:28.728955157 +0000 UTC m=+88.160810599" Jul 14 18:11:28.733634 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.733609 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-bound-sa-token\") pod \"21df3e31-0497-4f60-b03c-edf3cf852737\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " Jul 14 18:11:28.733791 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.733649 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-image-registry-private-configuration\") pod \"21df3e31-0497-4f60-b03c-edf3cf852737\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " Jul 14 18:11:28.733791 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.733691 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-kube-api-access-48fsv\") pod \"21df3e31-0497-4f60-b03c-edf3cf852737\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " Jul 14 18:11:28.733791 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.733728 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-installation-pull-secrets\") pod \"21df3e31-0497-4f60-b03c-edf3cf852737\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " Jul 14 18:11:28.733791 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.733759 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-trusted-ca\") pod \"21df3e31-0497-4f60-b03c-edf3cf852737\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " Jul 14 18:11:28.734007 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.733799 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-registry-certificates\") pod \"21df3e31-0497-4f60-b03c-edf3cf852737\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " Jul 14 18:11:28.734007 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.733825 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21df3e31-0497-4f60-b03c-edf3cf852737-ca-trust-extracted\") pod \"21df3e31-0497-4f60-b03c-edf3cf852737\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " Jul 14 18:11:28.734007 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.733896 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") pod \"21df3e31-0497-4f60-b03c-edf3cf852737\" (UID: \"21df3e31-0497-4f60-b03c-edf3cf852737\") " Jul 14 18:11:28.736336 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.734452 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "21df3e31-0497-4f60-b03c-edf3cf852737" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:11:28.736336 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.735234 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "21df3e31-0497-4f60-b03c-edf3cf852737" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:11:28.740032 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.739946 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "21df3e31-0497-4f60-b03c-edf3cf852737" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:11:28.740163 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.740121 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "21df3e31-0497-4f60-b03c-edf3cf852737" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:11:28.740305 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.740278 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "21df3e31-0497-4f60-b03c-edf3cf852737" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:11:28.740489 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.740441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "21df3e31-0497-4f60-b03c-edf3cf852737" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:11:28.743423 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.742835 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-kube-api-access-48fsv" (OuterVolumeSpecName: "kube-api-access-48fsv") pod "21df3e31-0497-4f60-b03c-edf3cf852737" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737"). InnerVolumeSpecName "kube-api-access-48fsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:11:28.752605 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.752500 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21df3e31-0497-4f60-b03c-edf3cf852737-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "21df3e31-0497-4f60-b03c-edf3cf852737" (UID: "21df3e31-0497-4f60-b03c-edf3cf852737"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:11:28.834802 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.834767 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-registry-certificates\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:11:28.834802 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.834801 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21df3e31-0497-4f60-b03c-edf3cf852737-ca-trust-extracted\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:11:28.835026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.834816 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-registry-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:11:28.835026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.834830 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-bound-sa-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:11:28.835026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.834843 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-image-registry-private-configuration\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:11:28.835026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.834856 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/21df3e31-0497-4f60-b03c-edf3cf852737-kube-api-access-48fsv\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:11:28.835026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.834871 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21df3e31-0497-4f60-b03c-edf3cf852737-installation-pull-secrets\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:11:28.835026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.834885 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21df3e31-0497-4f60-b03c-edf3cf852737-trusted-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:11:28.987266 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.987222 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-658f887d4f-cfwbv"] Jul 14 18:11:28.996438 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:28.996358 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-658f887d4f-cfwbv"] Jul 14 18:11:29.209475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:29.209434 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21df3e31-0497-4f60-b03c-edf3cf852737" path="/var/lib/kubelet/pods/21df3e31-0497-4f60-b03c-edf3cf852737/volumes" Jul 14 18:11:29.663989 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:29.663840 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdfnk" event={"ID":"48386c71-621c-42dd-b9f9-5be94e9f50ad","Type":"ContainerStarted","Data":"e8a54e69383e747479b05d986663482727f0757a1593fbeb175d14ce239000c3"} Jul 14 18:11:29.663989 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:29.663904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdfnk" event={"ID":"48386c71-621c-42dd-b9f9-5be94e9f50ad","Type":"ContainerStarted","Data":"6bfc08a3d5871f14d562d3dfdb3d7c663b362c158dae6bde8b2582e8a627453b"} Jul 14 18:11:29.688948 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:29.688833 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kdfnk" podStartSLOduration=3.458686724 podStartE2EDuration="4.688736259s" totalImagesPullingTime="1.152283784s" totalInitContainerRuntime="77.765751ms" isStatefulPod=true podCreationTimestamp="2026-07-14 18:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:29.687816043 +0000 UTC m=+89.119671479" watchObservedRunningTime="2026-07-14 18:11:29.688736259 +0000 UTC m=+89.120591695" Jul 14 18:11:32.687135 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:32.687097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k7vdr" event={"ID":"7ee50913-fbec-42f0-a1e8-55242e9c5ff7","Type":"ContainerStarted","Data":"1b954f0f75e1dbb516a9df3eaf8d032c38106c9484d6dc5592a79673798b4bd5"} Jul 14 18:11:32.712601 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:32.712515 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k7vdr" podStartSLOduration=34.033614286 podStartE2EDuration="53.712492754s" totalImagesPullingTime="19.678878468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:32.71135878 +0000 UTC m=+92.143214204" watchObservedRunningTime="2026-07-14 18:11:32.712492754 +0000 UTC m=+92.144348190" Jul 14 18:11:41.430146 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:41.430116 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678b44b47f-gl7cw"] Jul 14 18:11:44.936861 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:44.936836 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxkkc_88d206d5-a34a-4da9-8d97-2ccb5c1c707d/serve-healthcheck-canary/0.log" Jul 14 18:11:50.612352 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:11:50.612315 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wjkg4" Jul 14 18:12:06.450530 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.450484 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-678b44b47f-gl7cw" podUID="978f8353-4ae5-4b5b-a46a-c52d51055c63" containerName="console" containerID="cri-o://3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9" gracePeriod=15 Jul 14 18:12:06.730264 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.730230 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:12:06.815871 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.815836 2579 generic.go:356] "Generic (PLEG): container finished" podID="978f8353-4ae5-4b5b-a46a-c52d51055c63" containerID="3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9" exitCode=0 Jul 14 18:12:06.816026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.815887 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678b44b47f-gl7cw" Jul 14 18:12:06.816026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.815891 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678b44b47f-gl7cw" event={"ID":"978f8353-4ae5-4b5b-a46a-c52d51055c63","Type":"ContainerDied","Data":"3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9"} Jul 14 18:12:06.816026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.815972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678b44b47f-gl7cw" event={"ID":"978f8353-4ae5-4b5b-a46a-c52d51055c63","Type":"ContainerDied","Data":"4994d9a02645295dd544388a4a7225b54f4d4a159d3ef5a563930a0262ddd058"} Jul 14 18:12:06.816026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.816001 2579 scope.go:117] "RemoveContainer" containerID="3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9" Jul 14 18:12:06.823720 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.823701 2579 scope.go:117] "RemoveContainer" containerID="3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9" Jul 14 18:12:06.824010 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:12:06.823993 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9\": container with ID starting with 3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9 not found: ID does not exist" containerID="3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9" Jul 14 18:12:06.824063 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.824027 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9"} err="failed to get container status \"3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9\": rpc error: code = NotFound desc = could not find container \"3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9\": container with ID starting with 3f0cccfc8b26b8e521ecd545d979cf2324a63405f4aa1531df5bc62e7cbc51f9 not found: ID does not exist" Jul 14 18:12:06.829277 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829259 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-serving-cert\") pod \"978f8353-4ae5-4b5b-a46a-c52d51055c63\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " Jul 14 18:12:06.829352 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5tzx\" (UniqueName: \"kubernetes.io/projected/978f8353-4ae5-4b5b-a46a-c52d51055c63-kube-api-access-h5tzx\") pod \"978f8353-4ae5-4b5b-a46a-c52d51055c63\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " Jul 14 18:12:06.829352 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829340 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-oauth-serving-cert\") pod \"978f8353-4ae5-4b5b-a46a-c52d51055c63\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " Jul 14 18:12:06.829416 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829354 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-oauth-config\") pod \"978f8353-4ae5-4b5b-a46a-c52d51055c63\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " Jul 14 18:12:06.829416 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829378 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-config\") pod \"978f8353-4ae5-4b5b-a46a-c52d51055c63\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " Jul 14 18:12:06.829416 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829397 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-service-ca\") pod \"978f8353-4ae5-4b5b-a46a-c52d51055c63\" (UID: \"978f8353-4ae5-4b5b-a46a-c52d51055c63\") " Jul 14 18:12:06.829871 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829842 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "978f8353-4ae5-4b5b-a46a-c52d51055c63" (UID: "978f8353-4ae5-4b5b-a46a-c52d51055c63"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:12:06.829958 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829878 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-config" (OuterVolumeSpecName: "console-config") pod "978f8353-4ae5-4b5b-a46a-c52d51055c63" (UID: "978f8353-4ae5-4b5b-a46a-c52d51055c63"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:12:06.829958 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.829937 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-service-ca" (OuterVolumeSpecName: "service-ca") pod "978f8353-4ae5-4b5b-a46a-c52d51055c63" (UID: "978f8353-4ae5-4b5b-a46a-c52d51055c63"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:12:06.831485 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.831461 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "978f8353-4ae5-4b5b-a46a-c52d51055c63" (UID: "978f8353-4ae5-4b5b-a46a-c52d51055c63"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:12:06.831617 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.831546 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978f8353-4ae5-4b5b-a46a-c52d51055c63-kube-api-access-h5tzx" (OuterVolumeSpecName: "kube-api-access-h5tzx") pod "978f8353-4ae5-4b5b-a46a-c52d51055c63" (UID: "978f8353-4ae5-4b5b-a46a-c52d51055c63"). InnerVolumeSpecName "kube-api-access-h5tzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:12:06.831681 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.831641 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "978f8353-4ae5-4b5b-a46a-c52d51055c63" (UID: "978f8353-4ae5-4b5b-a46a-c52d51055c63"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:12:06.930090 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.930052 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-serving-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:12:06.930090 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.930085 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tzx\" (UniqueName: \"kubernetes.io/projected/978f8353-4ae5-4b5b-a46a-c52d51055c63-kube-api-access-h5tzx\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:12:06.930090 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.930094 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-oauth-serving-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:12:06.930317 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.930104 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-oauth-config\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:12:06.930317 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.930114 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-console-config\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:12:06.930317 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:06.930122 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/978f8353-4ae5-4b5b-a46a-c52d51055c63-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:12:07.143097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:07.143066 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678b44b47f-gl7cw"] Jul 14 18:12:07.149863 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:07.149841 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-678b44b47f-gl7cw"] Jul 14 18:12:07.205376 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:07.205348 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978f8353-4ae5-4b5b-a46a-c52d51055c63" path="/var/lib/kubelet/pods/978f8353-4ae5-4b5b-a46a-c52d51055c63/volumes" Jul 14 18:12:25.433144 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.433103 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd497ddcb-dntfv"] Jul 14 18:12:25.433749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.433450 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21df3e31-0497-4f60-b03c-edf3cf852737" containerName="registry" Jul 14 18:12:25.433749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.433468 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="21df3e31-0497-4f60-b03c-edf3cf852737" containerName="registry" Jul 14 18:12:25.433749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.433487 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978f8353-4ae5-4b5b-a46a-c52d51055c63" containerName="console" Jul 14 18:12:25.433749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.433496 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="978f8353-4ae5-4b5b-a46a-c52d51055c63" containerName="console" Jul 14 18:12:25.433749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.433589 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="978f8353-4ae5-4b5b-a46a-c52d51055c63" containerName="console" Jul 14 18:12:25.433749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.433616 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="21df3e31-0497-4f60-b03c-edf3cf852737" containerName="registry" Jul 14 18:12:25.436597 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.436577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.439572 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.439539 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 18:12:25.439669 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.439544 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 18:12:25.440860 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.440832 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 18:12:25.440974 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.440838 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 18:12:25.440974 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.440894 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 18:12:25.440974 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.440946 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z5ljq\"" Jul 14 18:12:25.445451 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.445432 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 18:12:25.448908 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.448884 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd497ddcb-dntfv"] Jul 14 18:12:25.570614 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.570578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-service-ca\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.570614 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.570623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-oauth-serving-cert\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.570854 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.570642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-console-config\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.570854 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.570663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlt9\" (UniqueName: \"kubernetes.io/projected/72ef0faf-0703-48d6-8842-455d42cea948-kube-api-access-htlt9\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.570854 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.570719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-serving-cert\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.570854 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.570743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-oauth-config\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.570854 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.570776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-trusted-ca-bundle\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.671864 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.671817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-service-ca\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.671864 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.671872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-oauth-serving-cert\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.672103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.671888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-console-config\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.672103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.671906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlt9\" (UniqueName: \"kubernetes.io/projected/72ef0faf-0703-48d6-8842-455d42cea948-kube-api-access-htlt9\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.672103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.671935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-serving-cert\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.672103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.671968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-oauth-config\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.672103 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.672022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-trusted-ca-bundle\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.672690 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.672667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-console-config\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.673122 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.672688 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-oauth-serving-cert\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.673288 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.672725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-service-ca\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.673396 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.673311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-trusted-ca-bundle\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.675005 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.674983 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-serving-cert\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.675171 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.675144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-oauth-config\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.683176 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.683123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlt9\" (UniqueName: \"kubernetes.io/projected/72ef0faf-0703-48d6-8842-455d42cea948-kube-api-access-htlt9\") pod \"console-5cd497ddcb-dntfv\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.745943 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.745910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:25.878402 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:25.878374 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd497ddcb-dntfv"] Jul 14 18:12:25.880929 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:12:25.880902 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ef0faf_0703_48d6_8842_455d42cea948.slice/crio-74d26a68299c5d3ba787db7167adafaa1fae7bd6a1941b81e8b0f409ee266dfc WatchSource:0}: Error finding container 74d26a68299c5d3ba787db7167adafaa1fae7bd6a1941b81e8b0f409ee266dfc: Status 404 returned error can't find the container with id 74d26a68299c5d3ba787db7167adafaa1fae7bd6a1941b81e8b0f409ee266dfc Jul 14 18:12:26.886449 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:26.886413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd497ddcb-dntfv" event={"ID":"72ef0faf-0703-48d6-8842-455d42cea948","Type":"ContainerStarted","Data":"cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd"} Jul 14 18:12:26.886840 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:26.886465 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd497ddcb-dntfv" event={"ID":"72ef0faf-0703-48d6-8842-455d42cea948","Type":"ContainerStarted","Data":"74d26a68299c5d3ba787db7167adafaa1fae7bd6a1941b81e8b0f409ee266dfc"} Jul 14 18:12:26.911647 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:26.911600 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cd497ddcb-dntfv" podStartSLOduration=1.9115831189999999 podStartE2EDuration="1.911583119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:12:26.909919943 +0000 UTC m=+146.341775376" watchObservedRunningTime="2026-07-14 18:12:26.911583119 +0000 UTC m=+146.343438552" Jul 14 18:12:35.746415 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:35.746378 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:35.746852 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:35.746442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:35.751022 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:35.750997 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:35.921453 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:35.921427 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:12:36.045066 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.044992 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bb66dd77f-5r68z"] Jul 14 18:12:36.048261 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.048244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.058476 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.058447 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb66dd77f-5r68z"] Jul 14 18:12:36.157452 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.157410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-serving-cert\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.157452 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.157451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-oauth-config\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.157677 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.157466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-trusted-ca-bundle\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.157677 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.157582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-console-config\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.157677 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.157630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpz2\" (UniqueName: \"kubernetes.io/projected/91363686-f027-461e-a47a-c1f5583d9d3d-kube-api-access-gjpz2\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.157677 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.157671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-service-ca\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.157804 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.157700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-oauth-serving-cert\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.258140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-service-ca\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.258140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-oauth-serving-cert\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.258384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-serving-cert\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.258384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-oauth-config\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.258384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-trusted-ca-bundle\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.258384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-console-config\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.258384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpz2\" (UniqueName: \"kubernetes.io/projected/91363686-f027-461e-a47a-c1f5583d9d3d-kube-api-access-gjpz2\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.258951 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-oauth-serving-cert\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.259054 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.258968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-service-ca\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.259054 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.259039 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-console-config\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.259139 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.259120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-trusted-ca-bundle\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.260730 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.260706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-serving-cert\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.260852 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.260799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-oauth-config\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.267705 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.267686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpz2\" (UniqueName: \"kubernetes.io/projected/91363686-f027-461e-a47a-c1f5583d9d3d-kube-api-access-gjpz2\") pod \"console-7bb66dd77f-5r68z\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.358809 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.358714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:36.489370 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.489334 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb66dd77f-5r68z"] Jul 14 18:12:36.491663 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:12:36.491630 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91363686_f027_461e_a47a_c1f5583d9d3d.slice/crio-d289ad4a146bdd56a20cea31d679153ae6ec5dbfbe624c6d746b5642bfc9392f WatchSource:0}: Error finding container d289ad4a146bdd56a20cea31d679153ae6ec5dbfbe624c6d746b5642bfc9392f: Status 404 returned error can't find the container with id d289ad4a146bdd56a20cea31d679153ae6ec5dbfbe624c6d746b5642bfc9392f Jul 14 18:12:36.927375 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.927325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb66dd77f-5r68z" event={"ID":"91363686-f027-461e-a47a-c1f5583d9d3d","Type":"ContainerStarted","Data":"b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9"} Jul 14 18:12:36.927878 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.927403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb66dd77f-5r68z" event={"ID":"91363686-f027-461e-a47a-c1f5583d9d3d","Type":"ContainerStarted","Data":"d289ad4a146bdd56a20cea31d679153ae6ec5dbfbe624c6d746b5642bfc9392f"} Jul 14 18:12:36.947384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:36.947317 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bb66dd77f-5r68z" podStartSLOduration=0.947298088 podStartE2EDuration="947.298088ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:12:36.946569549 +0000 UTC m=+156.378424975" watchObservedRunningTime="2026-07-14 18:12:36.947298088 +0000 UTC m=+156.379153523" Jul 14 18:12:46.359420 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:46.359373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:46.359831 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:46.359444 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:46.364223 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:46.364194 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:46.969968 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:46.969943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:12:47.034300 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:12:47.034270 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd497ddcb-dntfv"] Jul 14 18:13:12.053625 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.053578 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cd497ddcb-dntfv" podUID="72ef0faf-0703-48d6-8842-455d42cea948" containerName="console" containerID="cri-o://cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd" gracePeriod=15 Jul 14 18:13:12.295639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.295613 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:13:12.332331 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htlt9\" (UniqueName: \"kubernetes.io/projected/72ef0faf-0703-48d6-8842-455d42cea948-kube-api-access-htlt9\") pod \"72ef0faf-0703-48d6-8842-455d42cea948\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " Jul 14 18:13:12.332331 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332308 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-console-config\") pod \"72ef0faf-0703-48d6-8842-455d42cea948\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " Jul 14 18:13:12.332574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332353 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-oauth-config\") pod \"72ef0faf-0703-48d6-8842-455d42cea948\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " Jul 14 18:13:12.332574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332372 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-oauth-serving-cert\") pod \"72ef0faf-0703-48d6-8842-455d42cea948\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " Jul 14 18:13:12.332574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332391 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-serving-cert\") pod \"72ef0faf-0703-48d6-8842-455d42cea948\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " Jul 14 18:13:12.332574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332409 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-trusted-ca-bundle\") pod \"72ef0faf-0703-48d6-8842-455d42cea948\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " Jul 14 18:13:12.332574 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332435 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-service-ca\") pod \"72ef0faf-0703-48d6-8842-455d42cea948\" (UID: \"72ef0faf-0703-48d6-8842-455d42cea948\") " Jul 14 18:13:12.332843 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332816 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-console-config" (OuterVolumeSpecName: "console-config") pod "72ef0faf-0703-48d6-8842-455d42cea948" (UID: "72ef0faf-0703-48d6-8842-455d42cea948"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:13:12.332899 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.332878 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-service-ca" (OuterVolumeSpecName: "service-ca") pod "72ef0faf-0703-48d6-8842-455d42cea948" (UID: "72ef0faf-0703-48d6-8842-455d42cea948"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:13:12.333053 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.333032 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "72ef0faf-0703-48d6-8842-455d42cea948" (UID: "72ef0faf-0703-48d6-8842-455d42cea948"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:13:12.333115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.333028 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "72ef0faf-0703-48d6-8842-455d42cea948" (UID: "72ef0faf-0703-48d6-8842-455d42cea948"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:13:12.334506 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.334480 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ef0faf-0703-48d6-8842-455d42cea948-kube-api-access-htlt9" (OuterVolumeSpecName: "kube-api-access-htlt9") pod "72ef0faf-0703-48d6-8842-455d42cea948" (UID: "72ef0faf-0703-48d6-8842-455d42cea948"). InnerVolumeSpecName "kube-api-access-htlt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:13:12.334854 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.334823 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "72ef0faf-0703-48d6-8842-455d42cea948" (UID: "72ef0faf-0703-48d6-8842-455d42cea948"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:13:12.334961 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.334924 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "72ef0faf-0703-48d6-8842-455d42cea948" (UID: "72ef0faf-0703-48d6-8842-455d42cea948"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:13:12.433821 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.433779 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-console-config\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:13:12.433821 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.433813 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-oauth-config\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:13:12.433821 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.433827 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-oauth-serving-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:13:12.434056 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.433839 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72ef0faf-0703-48d6-8842-455d42cea948-console-serving-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:13:12.434056 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.433853 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-trusted-ca-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:13:12.434056 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.433866 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72ef0faf-0703-48d6-8842-455d42cea948-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:13:12.434056 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:12.433877 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlt9\" (UniqueName: \"kubernetes.io/projected/72ef0faf-0703-48d6-8842-455d42cea948-kube-api-access-htlt9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:13:13.060137 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.060105 2579 generic.go:356] "Generic (PLEG): container finished" podID="72ef0faf-0703-48d6-8842-455d42cea948" containerID="cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd" exitCode=0 Jul 14 18:13:13.060585 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.060152 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd497ddcb-dntfv" Jul 14 18:13:13.060585 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.060178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd497ddcb-dntfv" event={"ID":"72ef0faf-0703-48d6-8842-455d42cea948","Type":"ContainerDied","Data":"cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd"} Jul 14 18:13:13.060585 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.060217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd497ddcb-dntfv" event={"ID":"72ef0faf-0703-48d6-8842-455d42cea948","Type":"ContainerDied","Data":"74d26a68299c5d3ba787db7167adafaa1fae7bd6a1941b81e8b0f409ee266dfc"} Jul 14 18:13:13.060585 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.060237 2579 scope.go:117] "RemoveContainer" containerID="cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd" Jul 14 18:13:13.068697 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.068680 2579 scope.go:117] "RemoveContainer" containerID="cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd" Jul 14 18:13:13.068952 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:13:13.068934 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd\": container with ID starting with cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd not found: ID does not exist" containerID="cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd" Jul 14 18:13:13.068995 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.068962 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd"} err="failed to get container status \"cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd\": rpc error: code = NotFound desc = could not find container \"cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd\": container with ID starting with cb923986a7ff8eeea796ed3d4ff62b8390b2dfce3e0107ec633a61c7e7288ebd not found: ID does not exist" Jul 14 18:13:13.087084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.087053 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd497ddcb-dntfv"] Jul 14 18:13:13.097358 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.097334 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cd497ddcb-dntfv"] Jul 14 18:13:13.206336 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:13.206304 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ef0faf-0703-48d6-8842-455d42cea948" path="/var/lib/kubelet/pods/72ef0faf-0703-48d6-8842-455d42cea948/volumes" Jul 14 18:13:59.401858 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.401825 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn"] Jul 14 18:13:59.402280 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.402095 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ef0faf-0703-48d6-8842-455d42cea948" containerName="console" Jul 14 18:13:59.402280 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.402107 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ef0faf-0703-48d6-8842-455d42cea948" containerName="console" Jul 14 18:13:59.402280 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.402153 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ef0faf-0703-48d6-8842-455d42cea948" containerName="console" Jul 14 18:13:59.405067 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.405050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.407912 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.407892 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 18:13:59.409062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.409047 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h7jj6\"" Jul 14 18:13:59.409127 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.409048 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 18:13:59.423386 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.423361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn"] Jul 14 18:13:59.572463 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.572427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvqk8\" (UniqueName: \"kubernetes.io/projected/cc808777-4368-4dd9-a76b-6642ca9c6163-kube-api-access-gvqk8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.572463 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.572470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.572711 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.572498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.673457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.673381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.673457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.673431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.673631 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.673486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqk8\" (UniqueName: \"kubernetes.io/projected/cc808777-4368-4dd9-a76b-6642ca9c6163-kube-api-access-gvqk8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.673803 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.673784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.673841 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.673814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.684928 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.684904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqk8\" (UniqueName: \"kubernetes.io/projected/cc808777-4368-4dd9-a76b-6642ca9c6163-kube-api-access-gvqk8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.713920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.713881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:13:59.845940 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:13:59.845910 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn"] Jul 14 18:13:59.848847 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:13:59.848809 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc808777_4368_4dd9_a76b_6642ca9c6163.slice/crio-48fcd839c042fa5843bcc57484d72ea009e06bcb53c5950ecffd5eb991ae1122 WatchSource:0}: Error finding container 48fcd839c042fa5843bcc57484d72ea009e06bcb53c5950ecffd5eb991ae1122: Status 404 returned error can't find the container with id 48fcd839c042fa5843bcc57484d72ea009e06bcb53c5950ecffd5eb991ae1122 Jul 14 18:14:00.226020 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:00.225987 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" event={"ID":"cc808777-4368-4dd9-a76b-6642ca9c6163","Type":"ContainerStarted","Data":"48fcd839c042fa5843bcc57484d72ea009e06bcb53c5950ecffd5eb991ae1122"} Jul 14 18:14:05.243725 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:05.243630 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerID="2c6c798b01cd3aabe455cdeb176b5882c4f12bf33c235fc65528cd3078520563" exitCode=0 Jul 14 18:14:05.243725 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:05.243671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" event={"ID":"cc808777-4368-4dd9-a76b-6642ca9c6163","Type":"ContainerDied","Data":"2c6c798b01cd3aabe455cdeb176b5882c4f12bf33c235fc65528cd3078520563"} Jul 14 18:14:08.255903 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:08.255866 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerID="2113371bff3e64dbca89ec039999dbd2ac7e3dbb56a8576a6599e268979ff3ee" exitCode=0 Jul 14 18:14:08.256275 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:08.255953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" event={"ID":"cc808777-4368-4dd9-a76b-6642ca9c6163","Type":"ContainerDied","Data":"2113371bff3e64dbca89ec039999dbd2ac7e3dbb56a8576a6599e268979ff3ee"} Jul 14 18:14:16.287806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:16.287766 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerID="82a113d70a00f3c58a274d08d1fde40a4481c2ca6f14f023c44e644f5f7ed532" exitCode=0 Jul 14 18:14:16.288216 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:16.287829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" event={"ID":"cc808777-4368-4dd9-a76b-6642ca9c6163","Type":"ContainerDied","Data":"82a113d70a00f3c58a274d08d1fde40a4481c2ca6f14f023c44e644f5f7ed532"} Jul 14 18:14:17.418053 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.418022 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:14:17.519118 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.519079 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-bundle\") pod \"cc808777-4368-4dd9-a76b-6642ca9c6163\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " Jul 14 18:14:17.519315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.519160 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvqk8\" (UniqueName: \"kubernetes.io/projected/cc808777-4368-4dd9-a76b-6642ca9c6163-kube-api-access-gvqk8\") pod \"cc808777-4368-4dd9-a76b-6642ca9c6163\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " Jul 14 18:14:17.519315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.519187 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-util\") pod \"cc808777-4368-4dd9-a76b-6642ca9c6163\" (UID: \"cc808777-4368-4dd9-a76b-6642ca9c6163\") " Jul 14 18:14:17.519765 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.519739 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-bundle" (OuterVolumeSpecName: "bundle") pod "cc808777-4368-4dd9-a76b-6642ca9c6163" (UID: "cc808777-4368-4dd9-a76b-6642ca9c6163"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:14:17.521296 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.521274 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc808777-4368-4dd9-a76b-6642ca9c6163-kube-api-access-gvqk8" (OuterVolumeSpecName: "kube-api-access-gvqk8") pod "cc808777-4368-4dd9-a76b-6642ca9c6163" (UID: "cc808777-4368-4dd9-a76b-6642ca9c6163"). InnerVolumeSpecName "kube-api-access-gvqk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:14:17.524084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.524060 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-util" (OuterVolumeSpecName: "util") pod "cc808777-4368-4dd9-a76b-6642ca9c6163" (UID: "cc808777-4368-4dd9-a76b-6642ca9c6163"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:14:17.620026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.619931 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvqk8\" (UniqueName: \"kubernetes.io/projected/cc808777-4368-4dd9-a76b-6642ca9c6163-kube-api-access-gvqk8\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:17.620026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.619964 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-util\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:17.620026 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:17.619978 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc808777-4368-4dd9-a76b-6642ca9c6163-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:18.296150 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:18.296119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" event={"ID":"cc808777-4368-4dd9-a76b-6642ca9c6163","Type":"ContainerDied","Data":"48fcd839c042fa5843bcc57484d72ea009e06bcb53c5950ecffd5eb991ae1122"} Jul 14 18:14:18.296150 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:18.296147 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406rhfn" Jul 14 18:14:18.296150 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:18.296162 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48fcd839c042fa5843bcc57484d72ea009e06bcb53c5950ecffd5eb991ae1122" Jul 14 18:14:22.721386 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.721349 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx"] Jul 14 18:14:22.721890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.721612 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerName="pull" Jul 14 18:14:22.721890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.721624 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerName="pull" Jul 14 18:14:22.721890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.721640 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerName="util" Jul 14 18:14:22.721890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.721645 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerName="util" Jul 14 18:14:22.721890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.721655 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerName="extract" Jul 14 18:14:22.721890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.721662 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerName="extract" Jul 14 18:14:22.721890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.721701 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc808777-4368-4dd9-a76b-6642ca9c6163" containerName="extract" Jul 14 18:14:22.767029 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.766991 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx"] Jul 14 18:14:22.767182 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.767108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" Jul 14 18:14:22.770428 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.770403 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:14:22.770584 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.770520 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-z4jz7\"" Jul 14 18:14:22.770650 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.770633 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 18:14:22.856806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.856762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff5fc6c4-6e19-480b-882e-f1403f86853a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-f4wrx\" (UID: \"ff5fc6c4-6e19-480b-882e-f1403f86853a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" Jul 14 18:14:22.856806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.856808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9gpw\" (UniqueName: \"kubernetes.io/projected/ff5fc6c4-6e19-480b-882e-f1403f86853a-kube-api-access-n9gpw\") pod \"cert-manager-operator-controller-manager-55c444bf86-f4wrx\" (UID: \"ff5fc6c4-6e19-480b-882e-f1403f86853a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" Jul 14 18:14:22.957907 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.957875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff5fc6c4-6e19-480b-882e-f1403f86853a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-f4wrx\" (UID: \"ff5fc6c4-6e19-480b-882e-f1403f86853a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" Jul 14 18:14:22.958069 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.957912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9gpw\" (UniqueName: \"kubernetes.io/projected/ff5fc6c4-6e19-480b-882e-f1403f86853a-kube-api-access-n9gpw\") pod \"cert-manager-operator-controller-manager-55c444bf86-f4wrx\" (UID: \"ff5fc6c4-6e19-480b-882e-f1403f86853a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" Jul 14 18:14:22.958320 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.958302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff5fc6c4-6e19-480b-882e-f1403f86853a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-f4wrx\" (UID: \"ff5fc6c4-6e19-480b-882e-f1403f86853a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" Jul 14 18:14:22.969443 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:22.969419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9gpw\" (UniqueName: \"kubernetes.io/projected/ff5fc6c4-6e19-480b-882e-f1403f86853a-kube-api-access-n9gpw\") pod \"cert-manager-operator-controller-manager-55c444bf86-f4wrx\" (UID: \"ff5fc6c4-6e19-480b-882e-f1403f86853a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" Jul 14 18:14:23.075900 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:23.075809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" Jul 14 18:14:23.236811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:23.236763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx"] Jul 14 18:14:23.239267 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:14:23.239238 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5fc6c4_6e19_480b_882e_f1403f86853a.slice/crio-9496f0344de338d977015ccdf755082d04e4baf1659feabab9b17b16762d665f WatchSource:0}: Error finding container 9496f0344de338d977015ccdf755082d04e4baf1659feabab9b17b16762d665f: Status 404 returned error can't find the container with id 9496f0344de338d977015ccdf755082d04e4baf1659feabab9b17b16762d665f Jul 14 18:14:23.315724 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:23.315690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" event={"ID":"ff5fc6c4-6e19-480b-882e-f1403f86853a","Type":"ContainerStarted","Data":"9496f0344de338d977015ccdf755082d04e4baf1659feabab9b17b16762d665f"} Jul 14 18:14:25.325070 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:25.325030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" event={"ID":"ff5fc6c4-6e19-480b-882e-f1403f86853a","Type":"ContainerStarted","Data":"5ba8fa40ef0c54ce58cba0b2cfb08c2cad5a540c7ba479523dccd32ea0f454cb"} Jul 14 18:14:25.361081 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:25.361034 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-f4wrx" podStartSLOduration=1.419601594 podStartE2EDuration="3.361018772s" totalImagesPullingTime="1.941417178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:14:25.359207525 +0000 UTC m=+264.791062950" watchObservedRunningTime="2026-07-14 18:14:25.361018772 +0000 UTC m=+264.792874205" Jul 14 18:14:26.412267 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.412234 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx"] Jul 14 18:14:26.432898 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.432870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.436270 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.436247 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 18:14:26.445237 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.445210 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h7jj6\"" Jul 14 18:14:26.445652 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.445229 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 18:14:26.447671 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.447647 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx"] Jul 14 18:14:26.487121 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.487086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.487290 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.487158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stxh8\" (UniqueName: \"kubernetes.io/projected/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-kube-api-access-stxh8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.487290 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.487196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.588542 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.588499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stxh8\" (UniqueName: \"kubernetes.io/projected/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-kube-api-access-stxh8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.588542 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.588546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.588805 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.588623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.588991 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.588969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.589068 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.589002 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.604256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.604224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stxh8\" (UniqueName: \"kubernetes.io/projected/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-kube-api-access-stxh8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.746250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.746217 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:26.893051 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:26.892890 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx"] Jul 14 18:14:26.895725 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:14:26.895700 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7075633a_6e1f_4ba1_bcf0_a9e0db4cc493.slice/crio-64aef6aef1286d4b215b9b277be0193936245a094936e087a34d06b4f213c914 WatchSource:0}: Error finding container 64aef6aef1286d4b215b9b277be0193936245a094936e087a34d06b4f213c914: Status 404 returned error can't find the container with id 64aef6aef1286d4b215b9b277be0193936245a094936e087a34d06b4f213c914 Jul 14 18:14:27.334250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:27.334159 2579 generic.go:356] "Generic (PLEG): container finished" podID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerID="2a7b7541184c86559e918e1c053d3436f93e0ec35bac6d1538412232ca2eac65" exitCode=0 Jul 14 18:14:27.334402 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:27.334255 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" event={"ID":"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493","Type":"ContainerDied","Data":"2a7b7541184c86559e918e1c053d3436f93e0ec35bac6d1538412232ca2eac65"} Jul 14 18:14:27.334402 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:27.334306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" event={"ID":"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493","Type":"ContainerStarted","Data":"64aef6aef1286d4b215b9b277be0193936245a094936e087a34d06b4f213c914"} Jul 14 18:14:31.352813 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:31.352782 2579 generic.go:356] "Generic (PLEG): container finished" podID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerID="21aae90624356b81e41ac470e80f40190b9287187cca4335af635de3a705db6b" exitCode=0 Jul 14 18:14:31.353185 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:31.352847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" event={"ID":"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493","Type":"ContainerDied","Data":"21aae90624356b81e41ac470e80f40190b9287187cca4335af635de3a705db6b"} Jul 14 18:14:32.358883 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.358846 2579 generic.go:356] "Generic (PLEG): container finished" podID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerID="98ea520594394d3219bdc9aa73d889d6845679a059c37065862a10586be8b15c" exitCode=0 Jul 14 18:14:32.359284 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.358899 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" event={"ID":"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493","Type":"ContainerDied","Data":"98ea520594394d3219bdc9aa73d889d6845679a059c37065862a10586be8b15c"} Jul 14 18:14:32.700064 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.699975 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8x962"] Jul 14 18:14:32.703875 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.703852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" Jul 14 18:14:32.706733 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.706706 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zch8x\"" Jul 14 18:14:32.706870 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.706811 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 18:14:32.706870 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.706854 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 18:14:32.715900 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.715877 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8x962"] Jul 14 18:14:32.737503 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.737476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be69a256-2791-4c33-9084-a5bc5f01496c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8x962\" (UID: \"be69a256-2791-4c33-9084-a5bc5f01496c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" Jul 14 18:14:32.737645 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.737515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzc6k\" (UniqueName: \"kubernetes.io/projected/be69a256-2791-4c33-9084-a5bc5f01496c-kube-api-access-zzc6k\") pod \"cert-manager-cainjector-559cdd8c87-8x962\" (UID: \"be69a256-2791-4c33-9084-a5bc5f01496c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" Jul 14 18:14:32.838507 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.838459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzc6k\" (UniqueName: \"kubernetes.io/projected/be69a256-2791-4c33-9084-a5bc5f01496c-kube-api-access-zzc6k\") pod \"cert-manager-cainjector-559cdd8c87-8x962\" (UID: \"be69a256-2791-4c33-9084-a5bc5f01496c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" Jul 14 18:14:32.838673 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.838587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be69a256-2791-4c33-9084-a5bc5f01496c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8x962\" (UID: \"be69a256-2791-4c33-9084-a5bc5f01496c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" Jul 14 18:14:32.848980 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.848949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be69a256-2791-4c33-9084-a5bc5f01496c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8x962\" (UID: \"be69a256-2791-4c33-9084-a5bc5f01496c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" Jul 14 18:14:32.849161 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:32.849140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzc6k\" (UniqueName: \"kubernetes.io/projected/be69a256-2791-4c33-9084-a5bc5f01496c-kube-api-access-zzc6k\") pod \"cert-manager-cainjector-559cdd8c87-8x962\" (UID: \"be69a256-2791-4c33-9084-a5bc5f01496c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" Jul 14 18:14:33.012811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.012775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" Jul 14 18:14:33.142471 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.142446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8x962"] Jul 14 18:14:33.144786 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:14:33.144761 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe69a256_2791_4c33_9084_a5bc5f01496c.slice/crio-98f04f97360db9274952818b8673335812954cac12ef25cdea5e211cd8d21171 WatchSource:0}: Error finding container 98f04f97360db9274952818b8673335812954cac12ef25cdea5e211cd8d21171: Status 404 returned error can't find the container with id 98f04f97360db9274952818b8673335812954cac12ef25cdea5e211cd8d21171 Jul 14 18:14:33.363783 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.363699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" event={"ID":"be69a256-2791-4c33-9084-a5bc5f01496c","Type":"ContainerStarted","Data":"98f04f97360db9274952818b8673335812954cac12ef25cdea5e211cd8d21171"} Jul 14 18:14:33.482593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.482569 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:33.543647 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.543620 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stxh8\" (UniqueName: \"kubernetes.io/projected/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-kube-api-access-stxh8\") pod \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " Jul 14 18:14:33.543792 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.543653 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-bundle\") pod \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " Jul 14 18:14:33.543792 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.543686 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-util\") pod \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\" (UID: \"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493\") " Jul 14 18:14:33.544052 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.544028 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-bundle" (OuterVolumeSpecName: "bundle") pod "7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" (UID: "7075633a-6e1f-4ba1-bcf0-a9e0db4cc493"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:14:33.545753 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.545730 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-kube-api-access-stxh8" (OuterVolumeSpecName: "kube-api-access-stxh8") pod "7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" (UID: "7075633a-6e1f-4ba1-bcf0-a9e0db4cc493"). InnerVolumeSpecName "kube-api-access-stxh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:14:33.548617 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.548595 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-util" (OuterVolumeSpecName: "util") pod "7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" (UID: "7075633a-6e1f-4ba1-bcf0-a9e0db4cc493"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:14:33.644260 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.644182 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-util\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:33.644260 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.644208 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stxh8\" (UniqueName: \"kubernetes.io/projected/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-kube-api-access-stxh8\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:33.644260 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:33.644221 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7075633a-6e1f-4ba1-bcf0-a9e0db4cc493-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:34.370043 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:34.370012 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" Jul 14 18:14:34.370043 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:34.370025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsjxxx" event={"ID":"7075633a-6e1f-4ba1-bcf0-a9e0db4cc493","Type":"ContainerDied","Data":"64aef6aef1286d4b215b9b277be0193936245a094936e087a34d06b4f213c914"} Jul 14 18:14:34.370577 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:34.370074 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64aef6aef1286d4b215b9b277be0193936245a094936e087a34d06b4f213c914" Jul 14 18:14:36.380098 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:36.380063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" event={"ID":"be69a256-2791-4c33-9084-a5bc5f01496c","Type":"ContainerStarted","Data":"d7ef681dda528364e0bb7e8fe2394f67ce3871a14601f64236fb05079a709afb"} Jul 14 18:14:36.408827 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:36.408774 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8x962" podStartSLOduration=1.9967908840000002 podStartE2EDuration="4.408756127s" totalImagesPullingTime="2.411965243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:14:36.407726634 +0000 UTC m=+275.839582078" watchObservedRunningTime="2026-07-14 18:14:36.408756127 +0000 UTC m=+275.840611561" Jul 14 18:14:47.023870 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.023834 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp"] Jul 14 18:14:47.024249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.024105 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerName="util" Jul 14 18:14:47.024249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.024115 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerName="util" Jul 14 18:14:47.024249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.024123 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerName="extract" Jul 14 18:14:47.024249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.024129 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerName="extract" Jul 14 18:14:47.024249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.024144 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerName="pull" Jul 14 18:14:47.024249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.024150 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerName="pull" Jul 14 18:14:47.024249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.024205 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7075633a-6e1f-4ba1-bcf0-a9e0db4cc493" containerName="extract" Jul 14 18:14:47.027258 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.027240 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.030092 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.030072 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h7jj6\"" Jul 14 18:14:47.030508 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.030494 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 18:14:47.031344 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.031328 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 18:14:47.062836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.062802 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp"] Jul 14 18:14:47.151381 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.151339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7529l\" (UniqueName: \"kubernetes.io/projected/3139a8e6-3820-4b61-8a0c-1edb963885e6-kube-api-access-7529l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.151598 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.151427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.151598 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.151452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.252254 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.252215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.252254 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.252251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.252503 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.252288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7529l\" (UniqueName: \"kubernetes.io/projected/3139a8e6-3820-4b61-8a0c-1edb963885e6-kube-api-access-7529l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.252653 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.252633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.252726 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.252685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.265263 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.265232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7529l\" (UniqueName: \"kubernetes.io/projected/3139a8e6-3820-4b61-8a0c-1edb963885e6-kube-api-access-7529l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.336395 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.336290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:47.472324 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:47.472285 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp"] Jul 14 18:14:47.474479 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:14:47.474446 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3139a8e6_3820_4b61_8a0c_1edb963885e6.slice/crio-8abddd394862faa7f0168a7a888d4ad13233b568395c6c9fac28374e79450314 WatchSource:0}: Error finding container 8abddd394862faa7f0168a7a888d4ad13233b568395c6c9fac28374e79450314: Status 404 returned error can't find the container with id 8abddd394862faa7f0168a7a888d4ad13233b568395c6c9fac28374e79450314 Jul 14 18:14:48.435108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:48.435075 2579 generic.go:356] "Generic (PLEG): container finished" podID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerID="d431071ba1b2dc7f05c132029ee143310e936dc000157004bd02cfa17dce69ac" exitCode=0 Jul 14 18:14:48.435479 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:48.435165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" event={"ID":"3139a8e6-3820-4b61-8a0c-1edb963885e6","Type":"ContainerDied","Data":"d431071ba1b2dc7f05c132029ee143310e936dc000157004bd02cfa17dce69ac"} Jul 14 18:14:48.435479 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:48.435215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" event={"ID":"3139a8e6-3820-4b61-8a0c-1edb963885e6","Type":"ContainerStarted","Data":"8abddd394862faa7f0168a7a888d4ad13233b568395c6c9fac28374e79450314"} Jul 14 18:14:49.440774 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:49.440685 2579 generic.go:356] "Generic (PLEG): container finished" podID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerID="b6b9b0d3f98e1c69f824a1d9ad56445d0a7a87c306bd4a00c249732fa4860666" exitCode=0 Jul 14 18:14:49.440774 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:49.440752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" event={"ID":"3139a8e6-3820-4b61-8a0c-1edb963885e6","Type":"ContainerDied","Data":"b6b9b0d3f98e1c69f824a1d9ad56445d0a7a87c306bd4a00c249732fa4860666"} Jul 14 18:14:50.447363 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:50.447329 2579 generic.go:356] "Generic (PLEG): container finished" podID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerID="a0d5f8554b36972d5d30830495f41df08069147b4bfb60ec804794753f94f16a" exitCode=0 Jul 14 18:14:50.447761 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:50.447405 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" event={"ID":"3139a8e6-3820-4b61-8a0c-1edb963885e6","Type":"ContainerDied","Data":"a0d5f8554b36972d5d30830495f41df08069147b4bfb60ec804794753f94f16a"} Jul 14 18:14:51.575342 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.575316 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:14:51.689323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.689286 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7529l\" (UniqueName: \"kubernetes.io/projected/3139a8e6-3820-4b61-8a0c-1edb963885e6-kube-api-access-7529l\") pod \"3139a8e6-3820-4b61-8a0c-1edb963885e6\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " Jul 14 18:14:51.689323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.689338 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-bundle\") pod \"3139a8e6-3820-4b61-8a0c-1edb963885e6\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " Jul 14 18:14:51.689605 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.689364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-util\") pod \"3139a8e6-3820-4b61-8a0c-1edb963885e6\" (UID: \"3139a8e6-3820-4b61-8a0c-1edb963885e6\") " Jul 14 18:14:51.690100 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.690074 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-bundle" (OuterVolumeSpecName: "bundle") pod "3139a8e6-3820-4b61-8a0c-1edb963885e6" (UID: "3139a8e6-3820-4b61-8a0c-1edb963885e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:14:51.691445 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.691415 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3139a8e6-3820-4b61-8a0c-1edb963885e6-kube-api-access-7529l" (OuterVolumeSpecName: "kube-api-access-7529l") pod "3139a8e6-3820-4b61-8a0c-1edb963885e6" (UID: "3139a8e6-3820-4b61-8a0c-1edb963885e6"). InnerVolumeSpecName "kube-api-access-7529l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:14:51.695082 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.694993 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-util" (OuterVolumeSpecName: "util") pod "3139a8e6-3820-4b61-8a0c-1edb963885e6" (UID: "3139a8e6-3820-4b61-8a0c-1edb963885e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:14:51.790381 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.790325 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7529l\" (UniqueName: \"kubernetes.io/projected/3139a8e6-3820-4b61-8a0c-1edb963885e6-kube-api-access-7529l\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:51.790381 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.790373 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:51.790381 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:51.790383 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3139a8e6-3820-4b61-8a0c-1edb963885e6-util\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:14:52.456823 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:52.456780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" event={"ID":"3139a8e6-3820-4b61-8a0c-1edb963885e6","Type":"ContainerDied","Data":"8abddd394862faa7f0168a7a888d4ad13233b568395c6c9fac28374e79450314"} Jul 14 18:14:52.456968 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:52.456842 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8abddd394862faa7f0168a7a888d4ad13233b568395c6c9fac28374e79450314" Jul 14 18:14:52.456968 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:14:52.456800 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g9mp" Jul 14 18:15:01.018710 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:01.018684 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:15:01.021573 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:01.021533 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:15:01.021859 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:01.021828 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 18:15:05.087379 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.087346 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4"] Jul 14 18:15:05.092315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.087616 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerName="util" Jul 14 18:15:05.092315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.087627 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerName="util" Jul 14 18:15:05.092315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.087638 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerName="pull" Jul 14 18:15:05.092315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.087643 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerName="pull" Jul 14 18:15:05.092315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.087649 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerName="extract" Jul 14 18:15:05.092315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.087655 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerName="extract" Jul 14 18:15:05.092315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.087710 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3139a8e6-3820-4b61-8a0c-1edb963885e6" containerName="extract" Jul 14 18:15:05.093228 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.093212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.099150 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.099130 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 18:15:05.099281 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.099170 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-52hn2\"" Jul 14 18:15:05.099868 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.099843 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 18:15:05.100375 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.100360 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 18:15:05.102787 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.102772 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 18:15:05.177273 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.177239 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4"] Jul 14 18:15:05.194412 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.194389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b62b7fd6-cbb3-4fc4-acad-feee534701bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.194540 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.194418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b62b7fd6-cbb3-4fc4-acad-feee534701bc-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.194540 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.194450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b62b7fd6-cbb3-4fc4-acad-feee534701bc-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.194540 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.194487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spg5c\" (UniqueName: \"kubernetes.io/projected/b62b7fd6-cbb3-4fc4-acad-feee534701bc-kube-api-access-spg5c\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.194669 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.194543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b62b7fd6-cbb3-4fc4-acad-feee534701bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.295896 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.295858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b62b7fd6-cbb3-4fc4-acad-feee534701bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.295896 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.295896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b62b7fd6-cbb3-4fc4-acad-feee534701bc-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.296154 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.295929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b62b7fd6-cbb3-4fc4-acad-feee534701bc-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.296154 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.295946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spg5c\" (UniqueName: \"kubernetes.io/projected/b62b7fd6-cbb3-4fc4-acad-feee534701bc-kube-api-access-spg5c\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.296154 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.295968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b62b7fd6-cbb3-4fc4-acad-feee534701bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.296312 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.296289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b62b7fd6-cbb3-4fc4-acad-feee534701bc-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.298283 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.298255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b62b7fd6-cbb3-4fc4-acad-feee534701bc-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.298451 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.298428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b62b7fd6-cbb3-4fc4-acad-feee534701bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.298543 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.298528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b62b7fd6-cbb3-4fc4-acad-feee534701bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.309290 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.309271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spg5c\" (UniqueName: \"kubernetes.io/projected/b62b7fd6-cbb3-4fc4-acad-feee534701bc-kube-api-access-spg5c\") pod \"opendatahub-operator-controller-manager-557b7f646c-rsnn4\" (UID: \"b62b7fd6-cbb3-4fc4-acad-feee534701bc\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.403666 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.403576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:05.584294 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.584261 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4"] Jul 14 18:15:05.586938 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:15:05.586913 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62b7fd6_cbb3_4fc4_acad_feee534701bc.slice/crio-8ee945311c1dc9f3e3c1396d11dd524c5e321f02c91ee1b92d838bd666a1e515 WatchSource:0}: Error finding container 8ee945311c1dc9f3e3c1396d11dd524c5e321f02c91ee1b92d838bd666a1e515: Status 404 returned error can't find the container with id 8ee945311c1dc9f3e3c1396d11dd524c5e321f02c91ee1b92d838bd666a1e515 Jul 14 18:15:05.588728 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:05.588713 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:15:06.518216 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:06.518181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" event={"ID":"b62b7fd6-cbb3-4fc4-acad-feee534701bc","Type":"ContainerStarted","Data":"8ee945311c1dc9f3e3c1396d11dd524c5e321f02c91ee1b92d838bd666a1e515"} Jul 14 18:15:09.534413 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:09.534378 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" event={"ID":"b62b7fd6-cbb3-4fc4-acad-feee534701bc","Type":"ContainerStarted","Data":"450e264fc9ac78907cd41b32f02dbc5f4e8b495a241c38cb71bceeac53c293e6"} Jul 14 18:15:10.539464 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:10.539431 2579 generic.go:356] "Generic (PLEG): container finished" podID="b62b7fd6-cbb3-4fc4-acad-feee534701bc" containerID="450e264fc9ac78907cd41b32f02dbc5f4e8b495a241c38cb71bceeac53c293e6" exitCode=0 Jul 14 18:15:10.539881 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:10.539515 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" event={"ID":"b62b7fd6-cbb3-4fc4-acad-feee534701bc","Type":"ContainerDied","Data":"450e264fc9ac78907cd41b32f02dbc5f4e8b495a241c38cb71bceeac53c293e6"} Jul 14 18:15:11.545077 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:11.545043 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" event={"ID":"b62b7fd6-cbb3-4fc4-acad-feee534701bc","Type":"ContainerStarted","Data":"c748bb3699adbb25157f16893cba4ce54518cade0372d0626790a6a355dcf380"} Jul 14 18:15:11.545451 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:11.545158 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:11.583415 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:11.583364 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" podStartSLOduration=2.139255791 podStartE2EDuration="6.583346977s" totalImagesPullingTime="3.433066041s" totalInitContainerRuntime="1.011025145s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:11.581740774 +0000 UTC m=+311.013596209" watchObservedRunningTime="2026-07-14 18:15:11.583346977 +0000 UTC m=+311.015202411" Jul 14 18:15:22.551594 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:22.551541 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-rsnn4" Jul 14 18:15:26.073231 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.073195 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lzz2q"] Jul 14 18:15:26.076498 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.076481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:26.080126 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.080106 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 14 18:15:26.093592 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.093570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lzz2q"] Jul 14 18:15:26.153469 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.153422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f3c473-0e59-40bd-a209-9bd8e76b000f-tls-certs\") pod \"model-serving-api-54d89bd5fc-lzz2q\" (UID: \"b9f3c473-0e59-40bd-a209-9bd8e76b000f\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:26.153724 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.153504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x294\" (UniqueName: \"kubernetes.io/projected/b9f3c473-0e59-40bd-a209-9bd8e76b000f-kube-api-access-7x294\") pod \"model-serving-api-54d89bd5fc-lzz2q\" (UID: \"b9f3c473-0e59-40bd-a209-9bd8e76b000f\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:26.254780 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.254744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x294\" (UniqueName: \"kubernetes.io/projected/b9f3c473-0e59-40bd-a209-9bd8e76b000f-kube-api-access-7x294\") pod \"model-serving-api-54d89bd5fc-lzz2q\" (UID: \"b9f3c473-0e59-40bd-a209-9bd8e76b000f\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:26.254959 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.254800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f3c473-0e59-40bd-a209-9bd8e76b000f-tls-certs\") pod \"model-serving-api-54d89bd5fc-lzz2q\" (UID: \"b9f3c473-0e59-40bd-a209-9bd8e76b000f\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:26.257285 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.257255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f3c473-0e59-40bd-a209-9bd8e76b000f-tls-certs\") pod \"model-serving-api-54d89bd5fc-lzz2q\" (UID: \"b9f3c473-0e59-40bd-a209-9bd8e76b000f\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:26.271395 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.271367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x294\" (UniqueName: \"kubernetes.io/projected/b9f3c473-0e59-40bd-a209-9bd8e76b000f-kube-api-access-7x294\") pod \"model-serving-api-54d89bd5fc-lzz2q\" (UID: \"b9f3c473-0e59-40bd-a209-9bd8e76b000f\") " pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:26.387020 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.386937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:26.519545 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.519470 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lzz2q"] Jul 14 18:15:26.522108 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:15:26.522082 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f3c473_0e59_40bd_a209_9bd8e76b000f.slice/crio-418eeb1bf43de49b4e6b900c8843b22b2095f4b737a65d4b60d37f3c80fe3397 WatchSource:0}: Error finding container 418eeb1bf43de49b4e6b900c8843b22b2095f4b737a65d4b60d37f3c80fe3397: Status 404 returned error can't find the container with id 418eeb1bf43de49b4e6b900c8843b22b2095f4b737a65d4b60d37f3c80fe3397 Jul 14 18:15:26.606155 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:26.606119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" event={"ID":"b9f3c473-0e59-40bd-a209-9bd8e76b000f","Type":"ContainerStarted","Data":"418eeb1bf43de49b4e6b900c8843b22b2095f4b737a65d4b60d37f3c80fe3397"} Jul 14 18:15:29.621991 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:29.621949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" event={"ID":"b9f3c473-0e59-40bd-a209-9bd8e76b000f","Type":"ContainerStarted","Data":"1a9de261f025b9d55266f9f417e95ee66357cdb8f0045cfd55df2422722753be"} Jul 14 18:15:29.622436 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:29.622027 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:29.643390 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:29.643342 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" podStartSLOduration=1.404803386 podStartE2EDuration="3.643328529s" totalImagesPullingTime="2.238525143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:29.643098582 +0000 UTC m=+329.074954017" watchObservedRunningTime="2026-07-14 18:15:29.643328529 +0000 UTC m=+329.075183964" Jul 14 18:15:33.749833 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.749800 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547"] Jul 14 18:15:33.753089 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.753073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.762567 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.762533 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:15:33.763817 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.763800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-strzs\"" Jul 14 18:15:33.763871 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.763815 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 18:15:33.763975 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.763959 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 18:15:33.764041 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.764027 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 18:15:33.764100 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.764036 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 18:15:33.771407 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.771377 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547"] Jul 14 18:15:33.806027 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.805993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdbt\" (UniqueName: \"kubernetes.io/projected/5c80640c-f601-4495-b83b-5d60beb03730-kube-api-access-5xdbt\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.806217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.806043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c80640c-f601-4495-b83b-5d60beb03730-cert\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.806217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.806128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c80640c-f601-4495-b83b-5d60beb03730-manager-config\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.806217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.806191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c80640c-f601-4495-b83b-5d60beb03730-metrics-cert\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.906617 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.906546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c80640c-f601-4495-b83b-5d60beb03730-manager-config\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.906770 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.906645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c80640c-f601-4495-b83b-5d60beb03730-metrics-cert\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.906770 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.906663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdbt\" (UniqueName: \"kubernetes.io/projected/5c80640c-f601-4495-b83b-5d60beb03730-kube-api-access-5xdbt\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.906770 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.906694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c80640c-f601-4495-b83b-5d60beb03730-cert\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.907229 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.907204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c80640c-f601-4495-b83b-5d60beb03730-manager-config\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.909110 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.909087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c80640c-f601-4495-b83b-5d60beb03730-cert\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.909201 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.909160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c80640c-f601-4495-b83b-5d60beb03730-metrics-cert\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:33.919897 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:33.919875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdbt\" (UniqueName: \"kubernetes.io/projected/5c80640c-f601-4495-b83b-5d60beb03730-kube-api-access-5xdbt\") pod \"lws-controller-manager-55b7dd5b8c-bl547\" (UID: \"5c80640c-f601-4495-b83b-5d60beb03730\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:34.061662 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:34.061547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:34.198263 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:34.198238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547"] Jul 14 18:15:34.200285 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:15:34.200258 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c80640c_f601_4495_b83b_5d60beb03730.slice/crio-c0b1055e71b6dddac3966f6eeca3d6679b40831f498c3897b981129b5dc8b8bc WatchSource:0}: Error finding container c0b1055e71b6dddac3966f6eeca3d6679b40831f498c3897b981129b5dc8b8bc: Status 404 returned error can't find the container with id c0b1055e71b6dddac3966f6eeca3d6679b40831f498c3897b981129b5dc8b8bc Jul 14 18:15:34.642790 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:34.642755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" event={"ID":"5c80640c-f601-4495-b83b-5d60beb03730","Type":"ContainerStarted","Data":"c0b1055e71b6dddac3966f6eeca3d6679b40831f498c3897b981129b5dc8b8bc"} Jul 14 18:15:36.654391 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:36.654353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" event={"ID":"5c80640c-f601-4495-b83b-5d60beb03730","Type":"ContainerStarted","Data":"8bf786dab017f506cce5df5a3442dd0af6b491bbd5ad978bc8d75cbaa0b92bd9"} Jul 14 18:15:36.654884 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:36.654465 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:36.686547 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:36.686494 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" podStartSLOduration=2.243967061 podStartE2EDuration="3.686473896s" totalImagesPullingTime="1.442506835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:36.685071469 +0000 UTC m=+336.116926914" watchObservedRunningTime="2026-07-14 18:15:36.686473896 +0000 UTC m=+336.118329329" Jul 14 18:15:40.629217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:40.629186 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-lzz2q" Jul 14 18:15:42.508420 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.508377 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9"] Jul 14 18:15:42.516851 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.516830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.522325 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.522294 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 18:15:42.528702 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.528678 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dmq7h\"" Jul 14 18:15:42.529881 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.529857 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 18:15:42.530003 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.529926 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 18:15:42.531179 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.531158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9"] Jul 14 18:15:42.571841 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.571809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.571841 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.571844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.572062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.571860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.572062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.571927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.572062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.571956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.572062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.571980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.572062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.571997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.572062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.572026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfw6l\" (UniqueName: \"kubernetes.io/projected/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-kube-api-access-sfw6l\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.572062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.572054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.672402 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfw6l\" (UniqueName: \"kubernetes.io/projected/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-kube-api-access-sfw6l\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.672639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.672639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.672639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.672887 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.672887 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.672887 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.672887 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.673097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.672910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.673153 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.673089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.673382 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.673354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.673382 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.673367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.673535 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.673410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.673535 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.673527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.675037 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.675013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.675549 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.675527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.682636 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.682545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.682740 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.682719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfw6l\" (UniqueName: \"kubernetes.io/projected/b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4-kube-api-access-sfw6l\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6785s5b9\" (UID: \"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:42.827081 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:42.827004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:43.019649 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:43.019624 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9"] Jul 14 18:15:43.021351 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:15:43.021323 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8af0c89_bb5f_4fea_9f2e_9383b6cc9df4.slice/crio-99606e389915ebb3aabb96757ac23d1c812141aebc3307e76062c8137327c949 WatchSource:0}: Error finding container 99606e389915ebb3aabb96757ac23d1c812141aebc3307e76062c8137327c949: Status 404 returned error can't find the container with id 99606e389915ebb3aabb96757ac23d1c812141aebc3307e76062c8137327c949 Jul 14 18:15:43.687199 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:43.687164 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" event={"ID":"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4","Type":"ContainerStarted","Data":"99606e389915ebb3aabb96757ac23d1c812141aebc3307e76062c8137327c949"} Jul 14 18:15:45.969403 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:45.969357 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:15:45.969656 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:45.969458 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:15:45.969656 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:45.969485 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:15:46.701816 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:46.701777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" event={"ID":"b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4","Type":"ContainerStarted","Data":"85ce57d2a758c3afac506a44cb145438800c20dcbe72c5ebd33828fb606117d8"} Jul 14 18:15:46.729119 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:46.729063 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" podStartSLOduration=1.7834185059999998 podStartE2EDuration="4.729044955s" totalImagesPullingTime="2.945626449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:46.726854472 +0000 UTC m=+346.158709907" watchObservedRunningTime="2026-07-14 18:15:46.729044955 +0000 UTC m=+346.160900452" Jul 14 18:15:46.828096 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:46.828061 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:47.660701 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:47.660668 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-bl547" Jul 14 18:15:47.831791 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:47.831757 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:48.709793 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:48.709759 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:15:48.710794 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:15:48.710774 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6785s5b9" Jul 14 18:16:26.123513 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.123471 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-px9bm"] Jul 14 18:16:26.127120 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.127094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" Jul 14 18:16:26.130135 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.130109 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:16:26.130285 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.130271 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:16:26.130356 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.130125 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-57ntm\"" Jul 14 18:16:26.140950 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.140922 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-px9bm"] Jul 14 18:16:26.213284 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.213244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vsrh\" (UniqueName: \"kubernetes.io/projected/c84e206f-191b-4cc0-b003-9ce14e98567b-kube-api-access-9vsrh\") pod \"kuadrant-operator-catalog-px9bm\" (UID: \"c84e206f-191b-4cc0-b003-9ce14e98567b\") " pod="kuadrant-system/kuadrant-operator-catalog-px9bm" Jul 14 18:16:26.314075 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.314037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vsrh\" (UniqueName: \"kubernetes.io/projected/c84e206f-191b-4cc0-b003-9ce14e98567b-kube-api-access-9vsrh\") pod \"kuadrant-operator-catalog-px9bm\" (UID: \"c84e206f-191b-4cc0-b003-9ce14e98567b\") " pod="kuadrant-system/kuadrant-operator-catalog-px9bm" Jul 14 18:16:26.325133 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.325112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vsrh\" (UniqueName: \"kubernetes.io/projected/c84e206f-191b-4cc0-b003-9ce14e98567b-kube-api-access-9vsrh\") pod \"kuadrant-operator-catalog-px9bm\" (UID: \"c84e206f-191b-4cc0-b003-9ce14e98567b\") " pod="kuadrant-system/kuadrant-operator-catalog-px9bm" Jul 14 18:16:26.437051 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.436963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" Jul 14 18:16:26.483226 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.483195 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-px9bm"] Jul 14 18:16:26.574568 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.574534 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-px9bm"] Jul 14 18:16:26.576477 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:16:26.576445 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84e206f_191b_4cc0_b003_9ce14e98567b.slice/crio-40f39235e7133745d889d94c205f4aee205f1391632879120af5dce1546cf1e5 WatchSource:0}: Error finding container 40f39235e7133745d889d94c205f4aee205f1391632879120af5dce1546cf1e5: Status 404 returned error can't find the container with id 40f39235e7133745d889d94c205f4aee205f1391632879120af5dce1546cf1e5 Jul 14 18:16:26.696403 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.696310 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2q25z"] Jul 14 18:16:26.700689 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.700665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:26.710547 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.710518 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2q25z"] Jul 14 18:16:26.818668 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.818615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6q8\" (UniqueName: \"kubernetes.io/projected/d42ca5a6-1f6f-4f17-8394-75660bcd6a2d-kube-api-access-kb6q8\") pod \"kuadrant-operator-catalog-2q25z\" (UID: \"d42ca5a6-1f6f-4f17-8394-75660bcd6a2d\") " pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:26.877815 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.877780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" event={"ID":"c84e206f-191b-4cc0-b003-9ce14e98567b","Type":"ContainerStarted","Data":"40f39235e7133745d889d94c205f4aee205f1391632879120af5dce1546cf1e5"} Jul 14 18:16:26.919111 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.919077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6q8\" (UniqueName: \"kubernetes.io/projected/d42ca5a6-1f6f-4f17-8394-75660bcd6a2d-kube-api-access-kb6q8\") pod \"kuadrant-operator-catalog-2q25z\" (UID: \"d42ca5a6-1f6f-4f17-8394-75660bcd6a2d\") " pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:26.929051 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:26.929023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6q8\" (UniqueName: \"kubernetes.io/projected/d42ca5a6-1f6f-4f17-8394-75660bcd6a2d-kube-api-access-kb6q8\") pod \"kuadrant-operator-catalog-2q25z\" (UID: \"d42ca5a6-1f6f-4f17-8394-75660bcd6a2d\") " pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:27.010869 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:27.010826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:27.171193 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:27.171116 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2q25z"] Jul 14 18:16:27.173272 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:16:27.173247 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42ca5a6_1f6f_4f17_8394_75660bcd6a2d.slice/crio-ca2c3f5a4a5fc300cbef904e47baaa18da8600acc7e9687ce3a858f75a731a2f WatchSource:0}: Error finding container ca2c3f5a4a5fc300cbef904e47baaa18da8600acc7e9687ce3a858f75a731a2f: Status 404 returned error can't find the container with id ca2c3f5a4a5fc300cbef904e47baaa18da8600acc7e9687ce3a858f75a731a2f Jul 14 18:16:27.885189 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:27.885140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" event={"ID":"d42ca5a6-1f6f-4f17-8394-75660bcd6a2d","Type":"ContainerStarted","Data":"ca2c3f5a4a5fc300cbef904e47baaa18da8600acc7e9687ce3a858f75a731a2f"} Jul 14 18:16:28.891430 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:28.891396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" event={"ID":"c84e206f-191b-4cc0-b003-9ce14e98567b","Type":"ContainerStarted","Data":"9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76"} Jul 14 18:16:28.891854 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:28.891477 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" podUID="c84e206f-191b-4cc0-b003-9ce14e98567b" containerName="registry-server" containerID="cri-o://9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76" gracePeriod=2 Jul 14 18:16:28.893156 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:28.893128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" event={"ID":"d42ca5a6-1f6f-4f17-8394-75660bcd6a2d","Type":"ContainerStarted","Data":"3accbf65f4c3d6576b76324cf2b0de42b81ca797bec05443d88595d25319555a"} Jul 14 18:16:28.911200 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:28.911152 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" podStartSLOduration=0.698306199 podStartE2EDuration="2.911137429s" totalImagesPullingTime="2.21283123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:16:28.910738635 +0000 UTC m=+388.342594081" watchObservedRunningTime="2026-07-14 18:16:28.911137429 +0000 UTC m=+388.342992862" Jul 14 18:16:28.930973 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:28.930924 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" podStartSLOduration=1.3121792939999999 podStartE2EDuration="2.930907358s" totalImagesPullingTime="1.618728064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:16:28.929227982 +0000 UTC m=+388.361083416" watchObservedRunningTime="2026-07-14 18:16:28.930907358 +0000 UTC m=+388.362762814" Jul 14 18:16:29.132978 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.132947 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-px9bm_c84e206f-191b-4cc0-b003-9ce14e98567b/registry-server/0.log" Jul 14 18:16:29.135398 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.135381 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" Jul 14 18:16:29.240578 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.240525 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vsrh\" (UniqueName: \"kubernetes.io/projected/c84e206f-191b-4cc0-b003-9ce14e98567b-kube-api-access-9vsrh\") pod \"c84e206f-191b-4cc0-b003-9ce14e98567b\" (UID: \"c84e206f-191b-4cc0-b003-9ce14e98567b\") " Jul 14 18:16:29.242687 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.242665 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84e206f-191b-4cc0-b003-9ce14e98567b-kube-api-access-9vsrh" (OuterVolumeSpecName: "kube-api-access-9vsrh") pod "c84e206f-191b-4cc0-b003-9ce14e98567b" (UID: "c84e206f-191b-4cc0-b003-9ce14e98567b"). InnerVolumeSpecName "kube-api-access-9vsrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:16:29.341529 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.341433 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vsrh\" (UniqueName: \"kubernetes.io/projected/c84e206f-191b-4cc0-b003-9ce14e98567b-kube-api-access-9vsrh\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:29.898813 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.898784 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-px9bm_c84e206f-191b-4cc0-b003-9ce14e98567b/registry-server/0.log" Jul 14 18:16:29.899192 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.898824 2579 generic.go:356] "Generic (PLEG): container finished" podID="c84e206f-191b-4cc0-b003-9ce14e98567b" containerID="9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76" exitCode=1 Jul 14 18:16:29.899192 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.898876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" event={"ID":"c84e206f-191b-4cc0-b003-9ce14e98567b","Type":"ContainerDied","Data":"9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76"} Jul 14 18:16:29.899192 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.898910 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" Jul 14 18:16:29.899192 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.898920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-px9bm" event={"ID":"c84e206f-191b-4cc0-b003-9ce14e98567b","Type":"ContainerDied","Data":"40f39235e7133745d889d94c205f4aee205f1391632879120af5dce1546cf1e5"} Jul 14 18:16:29.899192 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.898940 2579 scope.go:117] "RemoveContainer" containerID="9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76" Jul 14 18:16:29.907277 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.907261 2579 scope.go:117] "RemoveContainer" containerID="9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76" Jul 14 18:16:29.907526 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:16:29.907510 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76\": container with ID starting with 9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76 not found: ID does not exist" containerID="9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76" Jul 14 18:16:29.907594 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.907534 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76"} err="failed to get container status \"9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76\": rpc error: code = NotFound desc = could not find container \"9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76\": container with ID starting with 9481cfd00c6298a7662d70e23c6ca96c4d7e2583ec70b9f1c035c8c198eabc76 not found: ID does not exist" Jul 14 18:16:29.921785 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.921755 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-px9bm"] Jul 14 18:16:29.924872 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:29.924851 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-px9bm"] Jul 14 18:16:31.206898 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:31.206858 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c84e206f-191b-4cc0-b003-9ce14e98567b" path="/var/lib/kubelet/pods/c84e206f-191b-4cc0-b003-9ce14e98567b/volumes" Jul 14 18:16:37.011720 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:37.011665 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:37.012296 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:37.011742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:37.034077 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:37.034047 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:37.954595 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:37.954570 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-2q25z" Jul 14 18:16:42.124245 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.124216 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q"] Jul 14 18:16:42.124646 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.124512 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84e206f-191b-4cc0-b003-9ce14e98567b" containerName="registry-server" Jul 14 18:16:42.124646 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.124523 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84e206f-191b-4cc0-b003-9ce14e98567b" containerName="registry-server" Jul 14 18:16:42.124646 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.124596 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84e206f-191b-4cc0-b003-9ce14e98567b" containerName="registry-server" Jul 14 18:16:42.128931 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.128914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.131666 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.131646 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n85pf\"" Jul 14 18:16:42.137345 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.137324 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q"] Jul 14 18:16:42.247222 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.247189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.247400 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.247238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkggm\" (UniqueName: \"kubernetes.io/projected/6a93c87e-3d84-4384-a32c-84e87fd39d42-kube-api-access-mkggm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.247400 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.247273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.348691 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.348643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkggm\" (UniqueName: \"kubernetes.io/projected/6a93c87e-3d84-4384-a32c-84e87fd39d42-kube-api-access-mkggm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.348798 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.348716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.348798 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.348765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.349198 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.349178 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.349234 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.349194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.359212 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.359178 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkggm\" (UniqueName: \"kubernetes.io/projected/6a93c87e-3d84-4384-a32c-84e87fd39d42-kube-api-access-mkggm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.438285 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.438204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:42.527709 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.527681 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn"] Jul 14 18:16:42.532303 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.532281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.545148 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.545114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn"] Jul 14 18:16:42.575538 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.575510 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q"] Jul 14 18:16:42.576870 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:16:42.576844 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a93c87e_3d84_4384_a32c_84e87fd39d42.slice/crio-2af419c73de0ee5948c68a5bde4859ccc2af00bdaa4db125eed8aee7a0438afa WatchSource:0}: Error finding container 2af419c73de0ee5948c68a5bde4859ccc2af00bdaa4db125eed8aee7a0438afa: Status 404 returned error can't find the container with id 2af419c73de0ee5948c68a5bde4859ccc2af00bdaa4db125eed8aee7a0438afa Jul 14 18:16:42.652092 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.652059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dp5v\" (UniqueName: \"kubernetes.io/projected/b1c1c286-bf1f-45ac-af37-d49925719e3a-kube-api-access-5dp5v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.652237 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.652105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.652237 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.652180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.753464 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.753426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.753664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.753497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dp5v\" (UniqueName: \"kubernetes.io/projected/b1c1c286-bf1f-45ac-af37-d49925719e3a-kube-api-access-5dp5v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.753664 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.753523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.753889 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.753867 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.753929 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.753900 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.764328 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.764304 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dp5v\" (UniqueName: \"kubernetes.io/projected/b1c1c286-bf1f-45ac-af37-d49925719e3a-kube-api-access-5dp5v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.842442 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.842406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:42.936353 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.936320 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7"] Jul 14 18:16:42.946278 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.946207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:42.958271 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.958241 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7"] Jul 14 18:16:42.959365 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.959335 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerID="797269038e5c0a0a0d8a394e943744d12639f39870afa56163345450994cd4ce" exitCode=0 Jul 14 18:16:42.959514 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.959434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" event={"ID":"6a93c87e-3d84-4384-a32c-84e87fd39d42","Type":"ContainerDied","Data":"797269038e5c0a0a0d8a394e943744d12639f39870afa56163345450994cd4ce"} Jul 14 18:16:42.959514 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.959476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" event={"ID":"6a93c87e-3d84-4384-a32c-84e87fd39d42","Type":"ContainerStarted","Data":"2af419c73de0ee5948c68a5bde4859ccc2af00bdaa4db125eed8aee7a0438afa"} Jul 14 18:16:42.980389 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:42.980359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn"] Jul 14 18:16:42.983287 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:16:42.983262 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c1c286_bf1f_45ac_af37_d49925719e3a.slice/crio-8372423dedb1ec82b93215b03c38a59bbc4fbd154710926f1af3d7ac08c9e7d5 WatchSource:0}: Error finding container 8372423dedb1ec82b93215b03c38a59bbc4fbd154710926f1af3d7ac08c9e7d5: Status 404 returned error can't find the container with id 8372423dedb1ec82b93215b03c38a59bbc4fbd154710926f1af3d7ac08c9e7d5 Jul 14 18:16:43.056949 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.056913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.057112 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.056975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhtk\" (UniqueName: \"kubernetes.io/projected/ef1c96dc-c802-40b9-933e-173d06bf9a13-kube-api-access-wlhtk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.057112 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.057003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.157815 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.157778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhtk\" (UniqueName: \"kubernetes.io/projected/ef1c96dc-c802-40b9-933e-173d06bf9a13-kube-api-access-wlhtk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.158277 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.157826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.158277 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.157879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.158384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.158316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.158384 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.158313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.168105 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.168072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhtk\" (UniqueName: \"kubernetes.io/projected/ef1c96dc-c802-40b9-933e-173d06bf9a13-kube-api-access-wlhtk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.259148 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.259110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:43.334829 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.334796 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl"] Jul 14 18:16:43.339877 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.339854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.350674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.350648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl"] Jul 14 18:16:43.400992 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.400967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7"] Jul 14 18:16:43.402301 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:16:43.402277 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1c96dc_c802_40b9_933e_173d06bf9a13.slice/crio-6de35da6ef9d0002255a69980aa6db001188cdffbdfbd46cb1ea609a296676eb WatchSource:0}: Error finding container 6de35da6ef9d0002255a69980aa6db001188cdffbdfbd46cb1ea609a296676eb: Status 404 returned error can't find the container with id 6de35da6ef9d0002255a69980aa6db001188cdffbdfbd46cb1ea609a296676eb Jul 14 18:16:43.460943 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.460919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hfd\" (UniqueName: \"kubernetes.io/projected/3d8796e6-2680-4c22-8b6d-4c77588c7676-kube-api-access-64hfd\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.461037 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.460963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.461037 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.460989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.562001 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.561909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hfd\" (UniqueName: \"kubernetes.io/projected/3d8796e6-2680-4c22-8b6d-4c77588c7676-kube-api-access-64hfd\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.562001 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.561966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.562001 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.561992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.562321 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.562299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.562360 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.562313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.571733 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.571711 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hfd\" (UniqueName: \"kubernetes.io/projected/3d8796e6-2680-4c22-8b6d-4c77588c7676-kube-api-access-64hfd\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.652849 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.652814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:43.784465 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.784383 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl"] Jul 14 18:16:43.786523 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:16:43.786500 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8796e6_2680_4c22_8b6d_4c77588c7676.slice/crio-7a58567bb90355028bcc87add1388159b71a3a8c5fb5c3db47c64090fc1d4365 WatchSource:0}: Error finding container 7a58567bb90355028bcc87add1388159b71a3a8c5fb5c3db47c64090fc1d4365: Status 404 returned error can't find the container with id 7a58567bb90355028bcc87add1388159b71a3a8c5fb5c3db47c64090fc1d4365 Jul 14 18:16:43.965998 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.965968 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerID="e9e3e1ece975e89537499504f44bfa917699805a818624cd6a47513eb9727031" exitCode=0 Jul 14 18:16:43.966116 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.966054 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" event={"ID":"3d8796e6-2680-4c22-8b6d-4c77588c7676","Type":"ContainerDied","Data":"e9e3e1ece975e89537499504f44bfa917699805a818624cd6a47513eb9727031"} Jul 14 18:16:43.966116 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.966110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" event={"ID":"3d8796e6-2680-4c22-8b6d-4c77588c7676","Type":"ContainerStarted","Data":"7a58567bb90355028bcc87add1388159b71a3a8c5fb5c3db47c64090fc1d4365"} Jul 14 18:16:43.967816 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.967790 2579 generic.go:356] "Generic (PLEG): container finished" podID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerID="23b20e40353b3edf85b26897e5fab4d4fc1bd62877128b8a1512d5453e5f30ce" exitCode=0 Jul 14 18:16:43.967988 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.967876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" event={"ID":"ef1c96dc-c802-40b9-933e-173d06bf9a13","Type":"ContainerDied","Data":"23b20e40353b3edf85b26897e5fab4d4fc1bd62877128b8a1512d5453e5f30ce"} Jul 14 18:16:43.967988 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.967921 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" event={"ID":"ef1c96dc-c802-40b9-933e-173d06bf9a13","Type":"ContainerStarted","Data":"6de35da6ef9d0002255a69980aa6db001188cdffbdfbd46cb1ea609a296676eb"} Jul 14 18:16:43.969782 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.969761 2579 generic.go:356] "Generic (PLEG): container finished" podID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerID="a5b622b0d2de27ecbebd1a3083ebadd6a90294527328c48b5b70b446184589ba" exitCode=0 Jul 14 18:16:43.969869 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.969812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" event={"ID":"b1c1c286-bf1f-45ac-af37-d49925719e3a","Type":"ContainerDied","Data":"a5b622b0d2de27ecbebd1a3083ebadd6a90294527328c48b5b70b446184589ba"} Jul 14 18:16:43.969869 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:43.969839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" event={"ID":"b1c1c286-bf1f-45ac-af37-d49925719e3a","Type":"ContainerStarted","Data":"8372423dedb1ec82b93215b03c38a59bbc4fbd154710926f1af3d7ac08c9e7d5"} Jul 14 18:16:44.979593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:44.979546 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerID="10ba4b56f0e81f71a92b232be334f526a2f445348d38c3228bc8ecaf99c1b857" exitCode=0 Jul 14 18:16:44.980013 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:44.979716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" event={"ID":"6a93c87e-3d84-4384-a32c-84e87fd39d42","Type":"ContainerDied","Data":"10ba4b56f0e81f71a92b232be334f526a2f445348d38c3228bc8ecaf99c1b857"} Jul 14 18:16:44.981480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:44.981460 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerID="06dfe4b3ae18fa6f00c8791124558233baac298f940614f527adc87d47ec4662" exitCode=0 Jul 14 18:16:44.981576 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:44.981509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" event={"ID":"3d8796e6-2680-4c22-8b6d-4c77588c7676","Type":"ContainerDied","Data":"06dfe4b3ae18fa6f00c8791124558233baac298f940614f527adc87d47ec4662"} Jul 14 18:16:45.987959 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:45.987925 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerID="0055f98186fe9a2ea7a030730bc4e875290a9fbc5f01cf03d4e93a5158294491" exitCode=0 Jul 14 18:16:45.988410 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:45.988013 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" event={"ID":"6a93c87e-3d84-4384-a32c-84e87fd39d42","Type":"ContainerDied","Data":"0055f98186fe9a2ea7a030730bc4e875290a9fbc5f01cf03d4e93a5158294491"} Jul 14 18:16:45.989741 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:45.989722 2579 generic.go:356] "Generic (PLEG): container finished" podID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerID="72b16ef50450d4869c43e497b420fa2adcaedbb0ded87a3295151c2d7509ce65" exitCode=0 Jul 14 18:16:45.989825 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:45.989801 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" event={"ID":"b1c1c286-bf1f-45ac-af37-d49925719e3a","Type":"ContainerDied","Data":"72b16ef50450d4869c43e497b420fa2adcaedbb0ded87a3295151c2d7509ce65"} Jul 14 18:16:45.991915 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:45.991892 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerID="5447bf4d6e00c65e482eb108853c502e6353075c2326d09d8828a78bf1e3b4b2" exitCode=0 Jul 14 18:16:45.992018 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:45.991925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" event={"ID":"3d8796e6-2680-4c22-8b6d-4c77588c7676","Type":"ContainerDied","Data":"5447bf4d6e00c65e482eb108853c502e6353075c2326d09d8828a78bf1e3b4b2"} Jul 14 18:16:47.001749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.001720 2579 generic.go:356] "Generic (PLEG): container finished" podID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerID="40bb979715ec2ed55b1607394125c06584946a64c06982103113d94028c4a113" exitCode=0 Jul 14 18:16:47.002210 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.001804 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" event={"ID":"b1c1c286-bf1f-45ac-af37-d49925719e3a","Type":"ContainerDied","Data":"40bb979715ec2ed55b1607394125c06584946a64c06982103113d94028c4a113"} Jul 14 18:16:47.213388 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.213367 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:47.216108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.216090 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:47.395447 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.395347 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-util\") pod \"3d8796e6-2680-4c22-8b6d-4c77588c7676\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " Jul 14 18:16:47.395447 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.395403 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-bundle\") pod \"6a93c87e-3d84-4384-a32c-84e87fd39d42\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " Jul 14 18:16:47.395704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.395465 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkggm\" (UniqueName: \"kubernetes.io/projected/6a93c87e-3d84-4384-a32c-84e87fd39d42-kube-api-access-mkggm\") pod \"6a93c87e-3d84-4384-a32c-84e87fd39d42\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " Jul 14 18:16:47.395704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.395541 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64hfd\" (UniqueName: \"kubernetes.io/projected/3d8796e6-2680-4c22-8b6d-4c77588c7676-kube-api-access-64hfd\") pod \"3d8796e6-2680-4c22-8b6d-4c77588c7676\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " Jul 14 18:16:47.395704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.395601 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-bundle\") pod \"3d8796e6-2680-4c22-8b6d-4c77588c7676\" (UID: \"3d8796e6-2680-4c22-8b6d-4c77588c7676\") " Jul 14 18:16:47.395704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.395636 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-util\") pod \"6a93c87e-3d84-4384-a32c-84e87fd39d42\" (UID: \"6a93c87e-3d84-4384-a32c-84e87fd39d42\") " Jul 14 18:16:47.396183 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.396156 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-bundle" (OuterVolumeSpecName: "bundle") pod "6a93c87e-3d84-4384-a32c-84e87fd39d42" (UID: "6a93c87e-3d84-4384-a32c-84e87fd39d42"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:16:47.396183 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.396165 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-bundle" (OuterVolumeSpecName: "bundle") pod "3d8796e6-2680-4c22-8b6d-4c77588c7676" (UID: "3d8796e6-2680-4c22-8b6d-4c77588c7676"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:16:47.397652 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.397632 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a93c87e-3d84-4384-a32c-84e87fd39d42-kube-api-access-mkggm" (OuterVolumeSpecName: "kube-api-access-mkggm") pod "6a93c87e-3d84-4384-a32c-84e87fd39d42" (UID: "6a93c87e-3d84-4384-a32c-84e87fd39d42"). InnerVolumeSpecName "kube-api-access-mkggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:16:47.398006 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.397986 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8796e6-2680-4c22-8b6d-4c77588c7676-kube-api-access-64hfd" (OuterVolumeSpecName: "kube-api-access-64hfd") pod "3d8796e6-2680-4c22-8b6d-4c77588c7676" (UID: "3d8796e6-2680-4c22-8b6d-4c77588c7676"). InnerVolumeSpecName "kube-api-access-64hfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:16:47.401834 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.401801 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-util" (OuterVolumeSpecName: "util") pod "6a93c87e-3d84-4384-a32c-84e87fd39d42" (UID: "6a93c87e-3d84-4384-a32c-84e87fd39d42"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:16:47.402011 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.401993 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-util" (OuterVolumeSpecName: "util") pod "3d8796e6-2680-4c22-8b6d-4c77588c7676" (UID: "3d8796e6-2680-4c22-8b6d-4c77588c7676"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:16:47.496495 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.496460 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hfd\" (UniqueName: \"kubernetes.io/projected/3d8796e6-2680-4c22-8b6d-4c77588c7676-kube-api-access-64hfd\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:47.496495 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.496491 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:47.496495 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.496500 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-util\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:47.496705 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.496508 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d8796e6-2680-4c22-8b6d-4c77588c7676-util\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:47.496705 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.496515 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a93c87e-3d84-4384-a32c-84e87fd39d42-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:47.496705 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:47.496523 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkggm\" (UniqueName: \"kubernetes.io/projected/6a93c87e-3d84-4384-a32c-84e87fd39d42-kube-api-access-mkggm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:48.009123 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.009095 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" Jul 14 18:16:48.009123 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.009106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q" event={"ID":"6a93c87e-3d84-4384-a32c-84e87fd39d42","Type":"ContainerDied","Data":"2af419c73de0ee5948c68a5bde4859ccc2af00bdaa4db125eed8aee7a0438afa"} Jul 14 18:16:48.009588 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.009153 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af419c73de0ee5948c68a5bde4859ccc2af00bdaa4db125eed8aee7a0438afa" Jul 14 18:16:48.011119 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.011095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" event={"ID":"3d8796e6-2680-4c22-8b6d-4c77588c7676","Type":"ContainerDied","Data":"7a58567bb90355028bcc87add1388159b71a3a8c5fb5c3db47c64090fc1d4365"} Jul 14 18:16:48.011242 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.011140 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a58567bb90355028bcc87add1388159b71a3a8c5fb5c3db47c64090fc1d4365" Jul 14 18:16:48.011242 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.011103 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl" Jul 14 18:16:48.012993 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.012969 2579 generic.go:356] "Generic (PLEG): container finished" podID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerID="74cb4ffd1c2abca646d18e43267217a830b57d7aa315c23843c4ab187eb55415" exitCode=0 Jul 14 18:16:48.013115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.013059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" event={"ID":"ef1c96dc-c802-40b9-933e-173d06bf9a13","Type":"ContainerDied","Data":"74cb4ffd1c2abca646d18e43267217a830b57d7aa315c23843c4ab187eb55415"} Jul 14 18:16:48.153502 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.153475 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:48.301667 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.301584 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dp5v\" (UniqueName: \"kubernetes.io/projected/b1c1c286-bf1f-45ac-af37-d49925719e3a-kube-api-access-5dp5v\") pod \"b1c1c286-bf1f-45ac-af37-d49925719e3a\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " Jul 14 18:16:48.301667 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.301621 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-bundle\") pod \"b1c1c286-bf1f-45ac-af37-d49925719e3a\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " Jul 14 18:16:48.301667 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.301659 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-util\") pod \"b1c1c286-bf1f-45ac-af37-d49925719e3a\" (UID: \"b1c1c286-bf1f-45ac-af37-d49925719e3a\") " Jul 14 18:16:48.302134 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.302104 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-bundle" (OuterVolumeSpecName: "bundle") pod "b1c1c286-bf1f-45ac-af37-d49925719e3a" (UID: "b1c1c286-bf1f-45ac-af37-d49925719e3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:16:48.303641 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.303620 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c1c286-bf1f-45ac-af37-d49925719e3a-kube-api-access-5dp5v" (OuterVolumeSpecName: "kube-api-access-5dp5v") pod "b1c1c286-bf1f-45ac-af37-d49925719e3a" (UID: "b1c1c286-bf1f-45ac-af37-d49925719e3a"). InnerVolumeSpecName "kube-api-access-5dp5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:16:48.309746 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.309717 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-util" (OuterVolumeSpecName: "util") pod "b1c1c286-bf1f-45ac-af37-d49925719e3a" (UID: "b1c1c286-bf1f-45ac-af37-d49925719e3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:16:48.402886 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.402851 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dp5v\" (UniqueName: \"kubernetes.io/projected/b1c1c286-bf1f-45ac-af37-d49925719e3a-kube-api-access-5dp5v\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:48.402886 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.402880 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:48.402886 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:48.402889 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1c1c286-bf1f-45ac-af37-d49925719e3a-util\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:49.019618 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:49.019586 2579 generic.go:356] "Generic (PLEG): container finished" podID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerID="3d7e9cd1e4a6a62c9b1b7ca33d583a729f9a1f35dd351c6964610aabf5c7a163" exitCode=0 Jul 14 18:16:49.020027 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:49.019672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" event={"ID":"ef1c96dc-c802-40b9-933e-173d06bf9a13","Type":"ContainerDied","Data":"3d7e9cd1e4a6a62c9b1b7ca33d583a729f9a1f35dd351c6964610aabf5c7a163"} Jul 14 18:16:49.021398 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:49.021374 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" Jul 14 18:16:49.021480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:49.021373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn" event={"ID":"b1c1c286-bf1f-45ac-af37-d49925719e3a","Type":"ContainerDied","Data":"8372423dedb1ec82b93215b03c38a59bbc4fbd154710926f1af3d7ac08c9e7d5"} Jul 14 18:16:49.021523 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:49.021496 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8372423dedb1ec82b93215b03c38a59bbc4fbd154710926f1af3d7ac08c9e7d5" Jul 14 18:16:50.163871 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.163847 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:50.318516 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.318423 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlhtk\" (UniqueName: \"kubernetes.io/projected/ef1c96dc-c802-40b9-933e-173d06bf9a13-kube-api-access-wlhtk\") pod \"ef1c96dc-c802-40b9-933e-173d06bf9a13\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " Jul 14 18:16:50.318516 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.318501 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-bundle\") pod \"ef1c96dc-c802-40b9-933e-173d06bf9a13\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " Jul 14 18:16:50.318717 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.318585 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-util\") pod \"ef1c96dc-c802-40b9-933e-173d06bf9a13\" (UID: \"ef1c96dc-c802-40b9-933e-173d06bf9a13\") " Jul 14 18:16:50.319057 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.319032 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-bundle" (OuterVolumeSpecName: "bundle") pod "ef1c96dc-c802-40b9-933e-173d06bf9a13" (UID: "ef1c96dc-c802-40b9-933e-173d06bf9a13"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:16:50.320539 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.320510 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1c96dc-c802-40b9-933e-173d06bf9a13-kube-api-access-wlhtk" (OuterVolumeSpecName: "kube-api-access-wlhtk") pod "ef1c96dc-c802-40b9-933e-173d06bf9a13" (UID: "ef1c96dc-c802-40b9-933e-173d06bf9a13"). InnerVolumeSpecName "kube-api-access-wlhtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:16:50.323097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.323055 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-util" (OuterVolumeSpecName: "util") pod "ef1c96dc-c802-40b9-933e-173d06bf9a13" (UID: "ef1c96dc-c802-40b9-933e-173d06bf9a13"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:16:50.419243 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.419211 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-util\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:50.419243 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.419238 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlhtk\" (UniqueName: \"kubernetes.io/projected/ef1c96dc-c802-40b9-933e-173d06bf9a13-kube-api-access-wlhtk\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:50.419243 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:50.419249 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef1c96dc-c802-40b9-933e-173d06bf9a13-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:16:51.032748 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:51.032711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" event={"ID":"ef1c96dc-c802-40b9-933e-173d06bf9a13","Type":"ContainerDied","Data":"6de35da6ef9d0002255a69980aa6db001188cdffbdfbd46cb1ea609a296676eb"} Jul 14 18:16:51.032913 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:51.032767 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de35da6ef9d0002255a69980aa6db001188cdffbdfbd46cb1ea609a296676eb" Jul 14 18:16:51.032913 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:51.032721 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7" Jul 14 18:16:58.161695 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.161665 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl"] Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.161949 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerName="util" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.161959 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerName="util" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.161967 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerName="extract" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.161974 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerName="extract" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.161984 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerName="util" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.161989 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerName="util" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162001 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerName="extract" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162006 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerName="extract" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162015 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerName="util" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162020 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerName="util" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162026 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerName="pull" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162030 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerName="pull" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162036 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerName="pull" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162043 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerName="pull" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162052 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerName="extract" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162057 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerName="extract" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162063 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerName="pull" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162067 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerName="pull" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162075 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerName="extract" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162080 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerName="extract" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162086 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerName="pull" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162091 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerName="pull" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162099 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerName="util" Jul 14 18:16:58.162108 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162104 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerName="util" Jul 14 18:16:58.162875 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162149 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a93c87e-3d84-4384-a32c-84e87fd39d42" containerName="extract" Jul 14 18:16:58.162875 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162156 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8796e6-2680-4c22-8b6d-4c77588c7676" containerName="extract" Jul 14 18:16:58.162875 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162161 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c1c286-bf1f-45ac-af37-d49925719e3a" containerName="extract" Jul 14 18:16:58.162875 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.162167 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef1c96dc-c802-40b9-933e-173d06bf9a13" containerName="extract" Jul 14 18:16:58.165178 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.165163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" Jul 14 18:16:58.167985 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.167962 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 14 18:16:58.168098 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.168042 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hx5rm\"" Jul 14 18:16:58.176158 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.176126 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl"] Jul 14 18:16:58.181901 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.181875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6px\" (UniqueName: \"kubernetes.io/projected/d48b7991-d77c-4705-96af-9754fadfebf3-kube-api-access-kf6px\") pod \"dns-operator-controller-manager-864dd78c4c-j4pfl\" (UID: \"d48b7991-d77c-4705-96af-9754fadfebf3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" Jul 14 18:16:58.282290 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.282241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6px\" (UniqueName: \"kubernetes.io/projected/d48b7991-d77c-4705-96af-9754fadfebf3-kube-api-access-kf6px\") pod \"dns-operator-controller-manager-864dd78c4c-j4pfl\" (UID: \"d48b7991-d77c-4705-96af-9754fadfebf3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" Jul 14 18:16:58.297117 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.297084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6px\" (UniqueName: \"kubernetes.io/projected/d48b7991-d77c-4705-96af-9754fadfebf3-kube-api-access-kf6px\") pod \"dns-operator-controller-manager-864dd78c4c-j4pfl\" (UID: \"d48b7991-d77c-4705-96af-9754fadfebf3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" Jul 14 18:16:58.475360 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.475333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" Jul 14 18:16:58.618646 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:58.618621 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl"] Jul 14 18:16:58.620707 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:16:58.620680 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48b7991_d77c_4705_96af_9754fadfebf3.slice/crio-e333beb86402fd2c33907b240ae3c185e13152ecab8b8e286aa60757d1f87eb8 WatchSource:0}: Error finding container e333beb86402fd2c33907b240ae3c185e13152ecab8b8e286aa60757d1f87eb8: Status 404 returned error can't find the container with id e333beb86402fd2c33907b240ae3c185e13152ecab8b8e286aa60757d1f87eb8 Jul 14 18:16:59.070356 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:16:59.070319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" event={"ID":"d48b7991-d77c-4705-96af-9754fadfebf3","Type":"ContainerStarted","Data":"e333beb86402fd2c33907b240ae3c185e13152ecab8b8e286aa60757d1f87eb8"} Jul 14 18:17:01.619426 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:01.619392 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n"] Jul 14 18:17:01.623488 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:01.623462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:01.626171 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:01.626147 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qmkr2\"" Jul 14 18:17:01.633785 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:01.633763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n"] Jul 14 18:17:01.708454 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:01.708307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5rz\" (UniqueName: \"kubernetes.io/projected/b049cd20-820c-4e4d-8980-4d03c335d365-kube-api-access-nl5rz\") pod \"limitador-operator-controller-manager-859d7b78bd-rq28n\" (UID: \"b049cd20-820c-4e4d-8980-4d03c335d365\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:01.809527 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:01.809489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5rz\" (UniqueName: \"kubernetes.io/projected/b049cd20-820c-4e4d-8980-4d03c335d365-kube-api-access-nl5rz\") pod \"limitador-operator-controller-manager-859d7b78bd-rq28n\" (UID: \"b049cd20-820c-4e4d-8980-4d03c335d365\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:01.819845 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:01.819812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5rz\" (UniqueName: \"kubernetes.io/projected/b049cd20-820c-4e4d-8980-4d03c335d365-kube-api-access-nl5rz\") pod \"limitador-operator-controller-manager-859d7b78bd-rq28n\" (UID: \"b049cd20-820c-4e4d-8980-4d03c335d365\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:01.936442 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:01.936345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:02.275359 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:02.275194 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n"] Jul 14 18:17:02.278660 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:17:02.278603 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb049cd20_820c_4e4d_8980_4d03c335d365.slice/crio-aa57394e71cf48a0e0e97aa49f4d78a031ede51eb8131fe4bd3f45d5f15394d7 WatchSource:0}: Error finding container aa57394e71cf48a0e0e97aa49f4d78a031ede51eb8131fe4bd3f45d5f15394d7: Status 404 returned error can't find the container with id aa57394e71cf48a0e0e97aa49f4d78a031ede51eb8131fe4bd3f45d5f15394d7 Jul 14 18:17:03.093472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:03.093428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" event={"ID":"b049cd20-820c-4e4d-8980-4d03c335d365","Type":"ContainerStarted","Data":"aa57394e71cf48a0e0e97aa49f4d78a031ede51eb8131fe4bd3f45d5f15394d7"} Jul 14 18:17:03.541363 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:03.541337 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 14 18:17:04.099939 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:04.099906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" event={"ID":"d48b7991-d77c-4705-96af-9754fadfebf3","Type":"ContainerStarted","Data":"a2fe084f7025c98bc693162ab9cb98992d6238a05d46e0d077e2e47ee6e8b79e"} Jul 14 18:17:04.100316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:04.100125 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" Jul 14 18:17:04.137572 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:04.137487 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" podStartSLOduration=1.222282447 podStartE2EDuration="6.13746858s" totalImagesPullingTime="4.915186133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:04.135876936 +0000 UTC m=+423.567732369" watchObservedRunningTime="2026-07-14 18:17:04.13746858 +0000 UTC m=+423.569324014" Jul 14 18:17:05.570237 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:05.570198 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q"] Jul 14 18:17:05.572285 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:05.572268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" Jul 14 18:17:05.575396 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:05.575372 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zjb8b\"" Jul 14 18:17:05.586599 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:05.586577 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q"] Jul 14 18:17:05.638265 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:05.638230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rq9\" (UniqueName: \"kubernetes.io/projected/6099eeb4-692c-4ee7-8908-6307b56aa223-kube-api-access-t7rq9\") pod \"authorino-operator-75cc8d5cb7-j9x7q\" (UID: \"6099eeb4-692c-4ee7-8908-6307b56aa223\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" Jul 14 18:17:05.738965 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:05.738929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rq9\" (UniqueName: \"kubernetes.io/projected/6099eeb4-692c-4ee7-8908-6307b56aa223-kube-api-access-t7rq9\") pod \"authorino-operator-75cc8d5cb7-j9x7q\" (UID: \"6099eeb4-692c-4ee7-8908-6307b56aa223\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" Jul 14 18:17:05.749291 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:05.749260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rq9\" (UniqueName: \"kubernetes.io/projected/6099eeb4-692c-4ee7-8908-6307b56aa223-kube-api-access-t7rq9\") pod \"authorino-operator-75cc8d5cb7-j9x7q\" (UID: \"6099eeb4-692c-4ee7-8908-6307b56aa223\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" Jul 14 18:17:05.882473 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:05.882390 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" Jul 14 18:17:06.029323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:06.029294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q"] Jul 14 18:17:06.030430 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:17:06.030407 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6099eeb4_692c_4ee7_8908_6307b56aa223.slice/crio-adbf749f6af62b9932e3f7906573d9296def73378c1555c1299c2acd97c9f07b WatchSource:0}: Error finding container adbf749f6af62b9932e3f7906573d9296def73378c1555c1299c2acd97c9f07b: Status 404 returned error can't find the container with id adbf749f6af62b9932e3f7906573d9296def73378c1555c1299c2acd97c9f07b Jul 14 18:17:06.112594 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:06.112539 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" event={"ID":"6099eeb4-692c-4ee7-8908-6307b56aa223","Type":"ContainerStarted","Data":"adbf749f6af62b9932e3f7906573d9296def73378c1555c1299c2acd97c9f07b"} Jul 14 18:17:07.122494 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:07.122400 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" event={"ID":"b049cd20-820c-4e4d-8980-4d03c335d365","Type":"ContainerStarted","Data":"03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316"} Jul 14 18:17:07.122934 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:07.122635 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:07.146368 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:07.146228 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" podStartSLOduration=1.620566154 podStartE2EDuration="6.146207083s" totalImagesPullingTime="4.525640929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:07.146191238 +0000 UTC m=+426.578046673" watchObservedRunningTime="2026-07-14 18:17:07.146207083 +0000 UTC m=+426.578062518" Jul 14 18:17:09.134787 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:09.134749 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" event={"ID":"6099eeb4-692c-4ee7-8908-6307b56aa223","Type":"ContainerStarted","Data":"9c5966c2e027fdfa1ae1468beddf577d55e0701ab7f353a2cc89c350f2c814c2"} Jul 14 18:17:09.135237 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:09.134890 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" Jul 14 18:17:09.170002 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:09.169942 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" podStartSLOduration=1.553875212 podStartE2EDuration="4.169927128s" totalImagesPullingTime="2.616051916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:09.160708031 +0000 UTC m=+428.592563464" watchObservedRunningTime="2026-07-14 18:17:09.169927128 +0000 UTC m=+428.601782562" Jul 14 18:17:15.108811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:15.108774 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pfl" Jul 14 18:17:18.131016 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:18.130979 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:20.141109 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:20.141080 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j9x7q" Jul 14 18:17:21.194970 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.194933 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn"] Jul 14 18:17:21.199691 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.199673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.202903 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.202873 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n85pf\"" Jul 14 18:17:21.202903 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.202890 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 14 18:17:21.203371 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.203352 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 14 18:17:21.213748 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.213721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn"] Jul 14 18:17:21.263604 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.263548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a4b8ab6-5d07-489d-96a7-c255b9115400-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.263783 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.263619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a4b8ab6-5d07-489d-96a7-c255b9115400-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.263783 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.263659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z25jz\" (UniqueName: \"kubernetes.io/projected/7a4b8ab6-5d07-489d-96a7-c255b9115400-kube-api-access-z25jz\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.364852 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.364818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a4b8ab6-5d07-489d-96a7-c255b9115400-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.365048 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.364860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a4b8ab6-5d07-489d-96a7-c255b9115400-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.365048 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.364896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z25jz\" (UniqueName: \"kubernetes.io/projected/7a4b8ab6-5d07-489d-96a7-c255b9115400-kube-api-access-z25jz\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.365583 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.365540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a4b8ab6-5d07-489d-96a7-c255b9115400-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.367321 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.367299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a4b8ab6-5d07-489d-96a7-c255b9115400-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.375834 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.375812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z25jz\" (UniqueName: \"kubernetes.io/projected/7a4b8ab6-5d07-489d-96a7-c255b9115400-kube-api-access-z25jz\") pod \"kuadrant-console-plugin-66b5774985-9gdvn\" (UID: \"7a4b8ab6-5d07-489d-96a7-c255b9115400\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.509874 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.509840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" Jul 14 18:17:21.644172 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:21.644147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn"] Jul 14 18:17:21.646038 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:17:21.646007 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a4b8ab6_5d07_489d_96a7_c255b9115400.slice/crio-36fa557e2deeb40f091b25b16b2e9f03825688e6c05f1c1ca2cb8ce569c0650e WatchSource:0}: Error finding container 36fa557e2deeb40f091b25b16b2e9f03825688e6c05f1c1ca2cb8ce569c0650e: Status 404 returned error can't find the container with id 36fa557e2deeb40f091b25b16b2e9f03825688e6c05f1c1ca2cb8ce569c0650e Jul 14 18:17:22.199028 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:22.198988 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" event={"ID":"7a4b8ab6-5d07-489d-96a7-c255b9115400","Type":"ContainerStarted","Data":"36fa557e2deeb40f091b25b16b2e9f03825688e6c05f1c1ca2cb8ce569c0650e"} Jul 14 18:17:31.366538 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.366458 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n"] Jul 14 18:17:31.366997 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.366733 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" containerName="manager" containerID="cri-o://03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316" gracePeriod=2 Jul 14 18:17:31.376652 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.376484 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n"] Jul 14 18:17:31.420752 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.420717 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7"] Jul 14 18:17:31.421225 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.421169 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" containerName="manager" Jul 14 18:17:31.421225 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.421191 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" containerName="manager" Jul 14 18:17:31.421424 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.421277 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" containerName="manager" Jul 14 18:17:31.424496 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.424476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" Jul 14 18:17:31.427128 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:17:31.427094 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rq28n\" is forbidden: User \"system:node:ip-10-0-129-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-146.ec2.internal' and this object" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:31.450226 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.450193 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7"] Jul 14 18:17:31.557779 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.557700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjp7k\" (UniqueName: \"kubernetes.io/projected/62bc3629-2691-4994-890b-95ce711820da-kube-api-access-gjp7k\") pod \"limitador-operator-controller-manager-859d7b78bd-tpxq7\" (UID: \"62bc3629-2691-4994-890b-95ce711820da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" Jul 14 18:17:31.622003 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.621947 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:31.624563 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:17:31.624520 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rq28n\" is forbidden: User \"system:node:ip-10-0-129-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-146.ec2.internal' and this object" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:31.659355 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.659301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjp7k\" (UniqueName: \"kubernetes.io/projected/62bc3629-2691-4994-890b-95ce711820da-kube-api-access-gjp7k\") pod \"limitador-operator-controller-manager-859d7b78bd-tpxq7\" (UID: \"62bc3629-2691-4994-890b-95ce711820da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" Jul 14 18:17:31.674651 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.674616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjp7k\" (UniqueName: \"kubernetes.io/projected/62bc3629-2691-4994-890b-95ce711820da-kube-api-access-gjp7k\") pod \"limitador-operator-controller-manager-859d7b78bd-tpxq7\" (UID: \"62bc3629-2691-4994-890b-95ce711820da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" Jul 14 18:17:31.737685 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.737645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" Jul 14 18:17:31.759941 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.759913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl5rz\" (UniqueName: \"kubernetes.io/projected/b049cd20-820c-4e4d-8980-4d03c335d365-kube-api-access-nl5rz\") pod \"b049cd20-820c-4e4d-8980-4d03c335d365\" (UID: \"b049cd20-820c-4e4d-8980-4d03c335d365\") " Jul 14 18:17:31.762191 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.762154 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b049cd20-820c-4e4d-8980-4d03c335d365-kube-api-access-nl5rz" (OuterVolumeSpecName: "kube-api-access-nl5rz") pod "b049cd20-820c-4e4d-8980-4d03c335d365" (UID: "b049cd20-820c-4e4d-8980-4d03c335d365"). InnerVolumeSpecName "kube-api-access-nl5rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:17:31.860531 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.860498 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl5rz\" (UniqueName: \"kubernetes.io/projected/b049cd20-820c-4e4d-8980-4d03c335d365-kube-api-access-nl5rz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:17:31.889114 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:31.889089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7"] Jul 14 18:17:31.890990 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:17:31.890960 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bc3629_2691_4994_890b_95ce711820da.slice/crio-c1e05ddc790c9f4b2af070b38d955446673e371526a5b3b0b1ebc03259816053 WatchSource:0}: Error finding container c1e05ddc790c9f4b2af070b38d955446673e371526a5b3b0b1ebc03259816053: Status 404 returned error can't find the container with id c1e05ddc790c9f4b2af070b38d955446673e371526a5b3b0b1ebc03259816053 Jul 14 18:17:32.258858 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:32.258772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" event={"ID":"62bc3629-2691-4994-890b-95ce711820da","Type":"ContainerStarted","Data":"3adabf66b2da1d8a0a989f2988b724e6b15a6f029de44ae90e68fe99534c24f7"} Jul 14 18:17:32.258858 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:32.258854 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" Jul 14 18:17:32.259119 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:32.258882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" event={"ID":"62bc3629-2691-4994-890b-95ce711820da","Type":"ContainerStarted","Data":"c1e05ddc790c9f4b2af070b38d955446673e371526a5b3b0b1ebc03259816053"} Jul 14 18:17:32.260349 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:32.260320 2579 generic.go:356] "Generic (PLEG): container finished" podID="b049cd20-820c-4e4d-8980-4d03c335d365" containerID="03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316" exitCode=0 Jul 14 18:17:32.260451 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:32.260376 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:32.260451 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:32.260416 2579 scope.go:117] "RemoveContainer" containerID="03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316" Jul 14 18:17:32.283155 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:32.283102 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" podStartSLOduration=1.283082365 podStartE2EDuration="1.283082365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:32.282071874 +0000 UTC m=+451.713927320" watchObservedRunningTime="2026-07-14 18:17:32.283082365 +0000 UTC m=+451.714937799" Jul 14 18:17:32.284374 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:17:32.284343 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rq28n\" is forbidden: User \"system:node:ip-10-0-129-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-146.ec2.internal' and this object" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:32.286679 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:17:32.286648 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rq28n\" is forbidden: User \"system:node:ip-10-0-129-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-146.ec2.internal' and this object" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rq28n" Jul 14 18:17:33.209155 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:33.209126 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b049cd20-820c-4e4d-8980-4d03c335d365" path="/var/lib/kubelet/pods/b049cd20-820c-4e4d-8980-4d03c335d365/volumes" Jul 14 18:17:43.270218 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:43.270183 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tpxq7" Jul 14 18:17:46.476700 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:46.476460 2579 scope.go:117] "RemoveContainer" containerID="03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316" Jul 14 18:17:46.476936 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:17:46.476799 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316\": container with ID starting with 03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316 not found: ID does not exist" containerID="03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316" Jul 14 18:17:46.476936 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:46.476829 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316"} err="failed to get container status \"03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316\": rpc error: code = NotFound desc = could not find container \"03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316\": container with ID starting with 03aafaa701220a8630636d624a17a90da133904235494031bd2aea2ecdf3a316 not found: ID does not exist" Jul 14 18:17:47.351624 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:47.351585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" event={"ID":"7a4b8ab6-5d07-489d-96a7-c255b9115400","Type":"ContainerStarted","Data":"9df0d70d63b92d4255f8f42a9212cee1b5e07071497f6552fa81bea99b76a39c"} Jul 14 18:17:47.372340 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:17:47.372285 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9gdvn" podStartSLOduration=1.464818099 podStartE2EDuration="26.372270365s" totalImagesPullingTime="24.907452266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:47.370080538 +0000 UTC m=+466.801935975" watchObservedRunningTime="2026-07-14 18:17:47.372270365 +0000 UTC m=+466.804125798" Jul 14 18:18:14.400694 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.400661 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wgz8w"] Jul 14 18:18:14.683697 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.683616 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgz8w"] Jul 14 18:18:14.683842 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.683776 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wgz8w" Jul 14 18:18:14.686778 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.686755 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6fvt7\"" Jul 14 18:18:14.722383 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.722338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tdv\" (UniqueName: \"kubernetes.io/projected/894e7ad4-1b7d-4a1e-8580-18e14f045aad-kube-api-access-w6tdv\") pod \"authorino-58cd955675-wgz8w\" (UID: \"894e7ad4-1b7d-4a1e-8580-18e14f045aad\") " pod="kuadrant-system/authorino-58cd955675-wgz8w" Jul 14 18:18:14.823533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.823495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tdv\" (UniqueName: \"kubernetes.io/projected/894e7ad4-1b7d-4a1e-8580-18e14f045aad-kube-api-access-w6tdv\") pod \"authorino-58cd955675-wgz8w\" (UID: \"894e7ad4-1b7d-4a1e-8580-18e14f045aad\") " pod="kuadrant-system/authorino-58cd955675-wgz8w" Jul 14 18:18:14.837223 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.837194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tdv\" (UniqueName: \"kubernetes.io/projected/894e7ad4-1b7d-4a1e-8580-18e14f045aad-kube-api-access-w6tdv\") pod \"authorino-58cd955675-wgz8w\" (UID: \"894e7ad4-1b7d-4a1e-8580-18e14f045aad\") " pod="kuadrant-system/authorino-58cd955675-wgz8w" Jul 14 18:18:14.856010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.855981 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-h45rf"] Jul 14 18:18:14.949006 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.948918 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h45rf"] Jul 14 18:18:14.949179 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.949033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h45rf" Jul 14 18:18:14.994851 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:14.994811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wgz8w" Jul 14 18:18:15.025251 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:15.025217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl4mp\" (UniqueName: \"kubernetes.io/projected/04d06fbd-38a3-4e3c-af7d-1f9767e8c04f-kube-api-access-vl4mp\") pod \"authorino-75d556bd77-h45rf\" (UID: \"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f\") " pod="kuadrant-system/authorino-75d556bd77-h45rf" Jul 14 18:18:15.125989 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:15.125958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl4mp\" (UniqueName: \"kubernetes.io/projected/04d06fbd-38a3-4e3c-af7d-1f9767e8c04f-kube-api-access-vl4mp\") pod \"authorino-75d556bd77-h45rf\" (UID: \"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f\") " pod="kuadrant-system/authorino-75d556bd77-h45rf" Jul 14 18:18:15.138478 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:15.138448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl4mp\" (UniqueName: \"kubernetes.io/projected/04d06fbd-38a3-4e3c-af7d-1f9767e8c04f-kube-api-access-vl4mp\") pod \"authorino-75d556bd77-h45rf\" (UID: \"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f\") " pod="kuadrant-system/authorino-75d556bd77-h45rf" Jul 14 18:18:15.148364 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:15.148336 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgz8w"] Jul 14 18:18:15.150357 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:18:15.150329 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod894e7ad4_1b7d_4a1e_8580_18e14f045aad.slice/crio-875ad0e57935b6355fe79bf76ce8a460871c4b658f33404930136accddaa3f6e WatchSource:0}: Error finding container 875ad0e57935b6355fe79bf76ce8a460871c4b658f33404930136accddaa3f6e: Status 404 returned error can't find the container with id 875ad0e57935b6355fe79bf76ce8a460871c4b658f33404930136accddaa3f6e Jul 14 18:18:15.258879 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:15.258841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h45rf" Jul 14 18:18:15.456204 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:15.456174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h45rf"] Jul 14 18:18:15.457430 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:18:15.457404 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04d06fbd_38a3_4e3c_af7d_1f9767e8c04f.slice/crio-ceb03dcb5b6a26f9fada790049ce83f92e80a673f03a89a48e9522bc168326bf WatchSource:0}: Error finding container ceb03dcb5b6a26f9fada790049ce83f92e80a673f03a89a48e9522bc168326bf: Status 404 returned error can't find the container with id ceb03dcb5b6a26f9fada790049ce83f92e80a673f03a89a48e9522bc168326bf Jul 14 18:18:15.496178 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:15.496135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h45rf" event={"ID":"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f","Type":"ContainerStarted","Data":"ceb03dcb5b6a26f9fada790049ce83f92e80a673f03a89a48e9522bc168326bf"} Jul 14 18:18:15.497368 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:15.497341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wgz8w" event={"ID":"894e7ad4-1b7d-4a1e-8580-18e14f045aad","Type":"ContainerStarted","Data":"875ad0e57935b6355fe79bf76ce8a460871c4b658f33404930136accddaa3f6e"} Jul 14 18:18:19.530170 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:19.530131 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h45rf" event={"ID":"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f","Type":"ContainerStarted","Data":"84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5"} Jul 14 18:18:19.531676 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:19.531653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wgz8w" event={"ID":"894e7ad4-1b7d-4a1e-8580-18e14f045aad","Type":"ContainerStarted","Data":"2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf"} Jul 14 18:18:19.551130 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:19.551084 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-h45rf" podStartSLOduration=2.078619325 podStartE2EDuration="5.551069793s" totalImagesPullingTime="3.472450468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:19.549912345 +0000 UTC m=+498.981767778" watchObservedRunningTime="2026-07-14 18:18:19.551069793 +0000 UTC m=+498.982925260" Jul 14 18:18:19.568845 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:19.568787 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wgz8w" podStartSLOduration=1.778159013 podStartE2EDuration="5.568766812s" totalImagesPullingTime="3.790607799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:19.566192381 +0000 UTC m=+498.998047816" watchObservedRunningTime="2026-07-14 18:18:19.568766812 +0000 UTC m=+499.000622247" Jul 14 18:18:19.599363 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:19.599327 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgz8w"] Jul 14 18:18:21.541482 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:21.541443 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wgz8w" podUID="894e7ad4-1b7d-4a1e-8580-18e14f045aad" containerName="authorino" containerID="cri-o://2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf" gracePeriod=30 Jul 14 18:18:21.787420 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:21.787386 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wgz8w" Jul 14 18:18:21.880542 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:21.880467 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6tdv\" (UniqueName: \"kubernetes.io/projected/894e7ad4-1b7d-4a1e-8580-18e14f045aad-kube-api-access-w6tdv\") pod \"894e7ad4-1b7d-4a1e-8580-18e14f045aad\" (UID: \"894e7ad4-1b7d-4a1e-8580-18e14f045aad\") " Jul 14 18:18:21.882462 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:21.882439 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894e7ad4-1b7d-4a1e-8580-18e14f045aad-kube-api-access-w6tdv" (OuterVolumeSpecName: "kube-api-access-w6tdv") pod "894e7ad4-1b7d-4a1e-8580-18e14f045aad" (UID: "894e7ad4-1b7d-4a1e-8580-18e14f045aad"). InnerVolumeSpecName "kube-api-access-w6tdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:18:21.981919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:21.981878 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tdv\" (UniqueName: \"kubernetes.io/projected/894e7ad4-1b7d-4a1e-8580-18e14f045aad-kube-api-access-w6tdv\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:18:22.548441 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.548408 2579 generic.go:356] "Generic (PLEG): container finished" podID="894e7ad4-1b7d-4a1e-8580-18e14f045aad" containerID="2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf" exitCode=0 Jul 14 18:18:22.548836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.548448 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wgz8w" Jul 14 18:18:22.548836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.548494 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wgz8w" event={"ID":"894e7ad4-1b7d-4a1e-8580-18e14f045aad","Type":"ContainerDied","Data":"2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf"} Jul 14 18:18:22.548836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.548531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wgz8w" event={"ID":"894e7ad4-1b7d-4a1e-8580-18e14f045aad","Type":"ContainerDied","Data":"875ad0e57935b6355fe79bf76ce8a460871c4b658f33404930136accddaa3f6e"} Jul 14 18:18:22.548836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.548547 2579 scope.go:117] "RemoveContainer" containerID="2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf" Jul 14 18:18:22.557478 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.557457 2579 scope.go:117] "RemoveContainer" containerID="2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf" Jul 14 18:18:22.557738 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:18:22.557718 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf\": container with ID starting with 2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf not found: ID does not exist" containerID="2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf" Jul 14 18:18:22.557809 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.557746 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf"} err="failed to get container status \"2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf\": rpc error: code = NotFound desc = could not find container \"2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf\": container with ID starting with 2620870705cc48f104a2f7d38154cb8cf8ae871779117e19d2f4f6a87bc08bcf not found: ID does not exist" Jul 14 18:18:22.574092 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.574066 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgz8w"] Jul 14 18:18:22.579646 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:22.579622 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wgz8w"] Jul 14 18:18:23.207022 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:23.206985 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894e7ad4-1b7d-4a1e-8580-18e14f045aad" path="/var/lib/kubelet/pods/894e7ad4-1b7d-4a1e-8580-18e14f045aad/volumes" Jul 14 18:18:44.850540 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:44.850499 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjfht"] Jul 14 18:18:44.851084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:44.850834 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894e7ad4-1b7d-4a1e-8580-18e14f045aad" containerName="authorino" Jul 14 18:18:44.851084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:44.850846 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="894e7ad4-1b7d-4a1e-8580-18e14f045aad" containerName="authorino" Jul 14 18:18:44.851084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:44.850907 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="894e7ad4-1b7d-4a1e-8580-18e14f045aad" containerName="authorino" Jul 14 18:18:44.853852 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:44.853834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjfht" Jul 14 18:18:44.861854 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:44.861828 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjfht"] Jul 14 18:18:44.981884 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:44.981844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxr7b\" (UniqueName: \"kubernetes.io/projected/4acecd60-5f69-4332-9b0a-2597b299f60c-kube-api-access-qxr7b\") pod \"authorino-5d4b67c446-sjfht\" (UID: \"4acecd60-5f69-4332-9b0a-2597b299f60c\") " pod="kuadrant-system/authorino-5d4b67c446-sjfht" Jul 14 18:18:45.082763 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.082730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr7b\" (UniqueName: \"kubernetes.io/projected/4acecd60-5f69-4332-9b0a-2597b299f60c-kube-api-access-qxr7b\") pod \"authorino-5d4b67c446-sjfht\" (UID: \"4acecd60-5f69-4332-9b0a-2597b299f60c\") " pod="kuadrant-system/authorino-5d4b67c446-sjfht" Jul 14 18:18:45.094764 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.094737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxr7b\" (UniqueName: \"kubernetes.io/projected/4acecd60-5f69-4332-9b0a-2597b299f60c-kube-api-access-qxr7b\") pod \"authorino-5d4b67c446-sjfht\" (UID: \"4acecd60-5f69-4332-9b0a-2597b299f60c\") " pod="kuadrant-system/authorino-5d4b67c446-sjfht" Jul 14 18:18:45.111093 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.111032 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjfht"] Jul 14 18:18:45.111258 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.111247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjfht" Jul 14 18:18:45.142949 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.142921 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b54dfbf98-z9vft"] Jul 14 18:18:45.147304 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.147288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" Jul 14 18:18:45.159201 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.159174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b54dfbf98-z9vft"] Jul 14 18:18:45.245691 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.245663 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjfht"] Jul 14 18:18:45.247340 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:18:45.247304 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4acecd60_5f69_4332_9b0a_2597b299f60c.slice/crio-d07fccba5265037a244e33398f1b0f45d541e8715da3ae5135ee81a08d3dc1c1 WatchSource:0}: Error finding container d07fccba5265037a244e33398f1b0f45d541e8715da3ae5135ee81a08d3dc1c1: Status 404 returned error can't find the container with id d07fccba5265037a244e33398f1b0f45d541e8715da3ae5135ee81a08d3dc1c1 Jul 14 18:18:45.284662 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.284634 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjpd9\" (UniqueName: \"kubernetes.io/projected/4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3-kube-api-access-mjpd9\") pod \"authorino-5b54dfbf98-z9vft\" (UID: \"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3\") " pod="kuadrant-system/authorino-5b54dfbf98-z9vft" Jul 14 18:18:45.385944 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.385859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjpd9\" (UniqueName: \"kubernetes.io/projected/4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3-kube-api-access-mjpd9\") pod \"authorino-5b54dfbf98-z9vft\" (UID: \"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3\") " pod="kuadrant-system/authorino-5b54dfbf98-z9vft" Jul 14 18:18:45.395924 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.395895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjpd9\" (UniqueName: \"kubernetes.io/projected/4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3-kube-api-access-mjpd9\") pod \"authorino-5b54dfbf98-z9vft\" (UID: \"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3\") " pod="kuadrant-system/authorino-5b54dfbf98-z9vft" Jul 14 18:18:45.457599 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.457532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" Jul 14 18:18:45.498397 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.498363 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b54dfbf98-z9vft"] Jul 14 18:18:45.534194 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.534165 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67b9fc9f46-zkrqg"] Jul 14 18:18:45.538288 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.538271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:18:45.541262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.541236 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 18:18:45.549494 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.549469 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67b9fc9f46-zkrqg"] Jul 14 18:18:45.591472 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.591446 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b54dfbf98-z9vft"] Jul 14 18:18:45.593315 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:18:45.593287 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4a4fa4_a245_4a10_bfee_b2ba86bdd5b3.slice/crio-eb6b9263cf309a13ff2edb6de6b4dbfada770a8b7c44a06029057a066fe91b4e WatchSource:0}: Error finding container eb6b9263cf309a13ff2edb6de6b4dbfada770a8b7c44a06029057a066fe91b4e: Status 404 returned error can't find the container with id eb6b9263cf309a13ff2edb6de6b4dbfada770a8b7c44a06029057a066fe91b4e Jul 14 18:18:45.667822 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.667735 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjfht" event={"ID":"4acecd60-5f69-4332-9b0a-2597b299f60c","Type":"ContainerStarted","Data":"d07fccba5265037a244e33398f1b0f45d541e8715da3ae5135ee81a08d3dc1c1"} Jul 14 18:18:45.668855 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.668830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" event={"ID":"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3","Type":"ContainerStarted","Data":"eb6b9263cf309a13ff2edb6de6b4dbfada770a8b7c44a06029057a066fe91b4e"} Jul 14 18:18:45.688247 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.688220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7926c46-9672-40aa-86ee-b51f8876492c-tls-cert\") pod \"authorino-67b9fc9f46-zkrqg\" (UID: \"a7926c46-9672-40aa-86ee-b51f8876492c\") " pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:18:45.688321 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.688255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jqq\" (UniqueName: \"kubernetes.io/projected/a7926c46-9672-40aa-86ee-b51f8876492c-kube-api-access-49jqq\") pod \"authorino-67b9fc9f46-zkrqg\" (UID: \"a7926c46-9672-40aa-86ee-b51f8876492c\") " pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:18:45.789122 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.789090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7926c46-9672-40aa-86ee-b51f8876492c-tls-cert\") pod \"authorino-67b9fc9f46-zkrqg\" (UID: \"a7926c46-9672-40aa-86ee-b51f8876492c\") " pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:18:45.789317 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.789139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jqq\" (UniqueName: \"kubernetes.io/projected/a7926c46-9672-40aa-86ee-b51f8876492c-kube-api-access-49jqq\") pod \"authorino-67b9fc9f46-zkrqg\" (UID: \"a7926c46-9672-40aa-86ee-b51f8876492c\") " pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:18:45.791562 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.791528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7926c46-9672-40aa-86ee-b51f8876492c-tls-cert\") pod \"authorino-67b9fc9f46-zkrqg\" (UID: \"a7926c46-9672-40aa-86ee-b51f8876492c\") " pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:18:45.798834 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.798814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jqq\" (UniqueName: \"kubernetes.io/projected/a7926c46-9672-40aa-86ee-b51f8876492c-kube-api-access-49jqq\") pod \"authorino-67b9fc9f46-zkrqg\" (UID: \"a7926c46-9672-40aa-86ee-b51f8876492c\") " pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:18:45.850677 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.850650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:18:45.984587 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:45.984564 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67b9fc9f46-zkrqg"] Jul 14 18:18:45.986573 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:18:45.986524 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7926c46_9672_40aa_86ee_b51f8876492c.slice/crio-af26c4ed09108f3e70d04d02f1b4903816d182c47d9376475f0f73861ec21e8e WatchSource:0}: Error finding container af26c4ed09108f3e70d04d02f1b4903816d182c47d9376475f0f73861ec21e8e: Status 404 returned error can't find the container with id af26c4ed09108f3e70d04d02f1b4903816d182c47d9376475f0f73861ec21e8e Jul 14 18:18:46.675648 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.675608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" event={"ID":"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3","Type":"ContainerStarted","Data":"e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b"} Jul 14 18:18:46.675970 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.675672 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" podUID="4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3" containerName="authorino" containerID="cri-o://e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b" gracePeriod=30 Jul 14 18:18:46.677218 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.677190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" event={"ID":"a7926c46-9672-40aa-86ee-b51f8876492c","Type":"ContainerStarted","Data":"351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db"} Jul 14 18:18:46.677218 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.677222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" event={"ID":"a7926c46-9672-40aa-86ee-b51f8876492c","Type":"ContainerStarted","Data":"af26c4ed09108f3e70d04d02f1b4903816d182c47d9376475f0f73861ec21e8e"} Jul 14 18:18:46.694484 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.694438 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" podStartSLOduration=1.273957263 podStartE2EDuration="1.694423726s" totalImagesPullingTime="420.466463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:46.69313275 +0000 UTC m=+526.124988207" watchObservedRunningTime="2026-07-14 18:18:46.694423726 +0000 UTC m=+526.126279181" Jul 14 18:18:46.711539 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.711486 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" podStartSLOduration=1.217636542 podStartE2EDuration="1.711468662s" totalImagesPullingTime="493.83212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:46.710213566 +0000 UTC m=+526.142069001" watchObservedRunningTime="2026-07-14 18:18:46.711468662 +0000 UTC m=+526.143324096" Jul 14 18:18:46.750048 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.749976 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h45rf"] Jul 14 18:18:46.750212 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.750144 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-h45rf" podUID="04d06fbd-38a3-4e3c-af7d-1f9767e8c04f" containerName="authorino" containerID="cri-o://84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5" gracePeriod=30 Jul 14 18:18:46.995166 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:46.995140 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" Jul 14 18:18:47.008435 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.008410 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h45rf" Jul 14 18:18:47.099352 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.099316 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl4mp\" (UniqueName: \"kubernetes.io/projected/04d06fbd-38a3-4e3c-af7d-1f9767e8c04f-kube-api-access-vl4mp\") pod \"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f\" (UID: \"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f\") " Jul 14 18:18:47.099548 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.099413 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjpd9\" (UniqueName: \"kubernetes.io/projected/4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3-kube-api-access-mjpd9\") pod \"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3\" (UID: \"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3\") " Jul 14 18:18:47.101620 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.101586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d06fbd-38a3-4e3c-af7d-1f9767e8c04f-kube-api-access-vl4mp" (OuterVolumeSpecName: "kube-api-access-vl4mp") pod "04d06fbd-38a3-4e3c-af7d-1f9767e8c04f" (UID: "04d06fbd-38a3-4e3c-af7d-1f9767e8c04f"). InnerVolumeSpecName "kube-api-access-vl4mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:18:47.101725 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.101699 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3-kube-api-access-mjpd9" (OuterVolumeSpecName: "kube-api-access-mjpd9") pod "4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3" (UID: "4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3"). InnerVolumeSpecName "kube-api-access-mjpd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:18:47.200519 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.200480 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjpd9\" (UniqueName: \"kubernetes.io/projected/4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3-kube-api-access-mjpd9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:18:47.200519 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.200510 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl4mp\" (UniqueName: \"kubernetes.io/projected/04d06fbd-38a3-4e3c-af7d-1f9767e8c04f-kube-api-access-vl4mp\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:18:47.683593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.683532 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3" containerID="e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b" exitCode=0 Jul 14 18:18:47.683593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.683592 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" Jul 14 18:18:47.683812 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.683623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" event={"ID":"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3","Type":"ContainerDied","Data":"e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b"} Jul 14 18:18:47.683812 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.683668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b54dfbf98-z9vft" event={"ID":"4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3","Type":"ContainerDied","Data":"eb6b9263cf309a13ff2edb6de6b4dbfada770a8b7c44a06029057a066fe91b4e"} Jul 14 18:18:47.683812 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.683690 2579 scope.go:117] "RemoveContainer" containerID="e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b" Jul 14 18:18:47.685102 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.685075 2579 generic.go:356] "Generic (PLEG): container finished" podID="04d06fbd-38a3-4e3c-af7d-1f9767e8c04f" containerID="84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5" exitCode=0 Jul 14 18:18:47.685210 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.685126 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h45rf" Jul 14 18:18:47.685210 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.685166 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h45rf" event={"ID":"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f","Type":"ContainerDied","Data":"84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5"} Jul 14 18:18:47.685210 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.685198 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h45rf" event={"ID":"04d06fbd-38a3-4e3c-af7d-1f9767e8c04f","Type":"ContainerDied","Data":"ceb03dcb5b6a26f9fada790049ce83f92e80a673f03a89a48e9522bc168326bf"} Jul 14 18:18:47.692479 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.692455 2579 scope.go:117] "RemoveContainer" containerID="e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b" Jul 14 18:18:47.692802 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:18:47.692777 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b\": container with ID starting with e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b not found: ID does not exist" containerID="e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b" Jul 14 18:18:47.692880 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.692813 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b"} err="failed to get container status \"e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b\": rpc error: code = NotFound desc = could not find container \"e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b\": container with ID starting with e141ff722d0a570dacf6452e359b3621accce743ac0c384ce5de0da76c854a6b not found: ID does not exist" Jul 14 18:18:47.692880 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.692836 2579 scope.go:117] "RemoveContainer" containerID="84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5" Jul 14 18:18:47.700506 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.700490 2579 scope.go:117] "RemoveContainer" containerID="84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5" Jul 14 18:18:47.700758 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:18:47.700736 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5\": container with ID starting with 84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5 not found: ID does not exist" containerID="84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5" Jul 14 18:18:47.700828 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.700762 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5"} err="failed to get container status \"84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5\": rpc error: code = NotFound desc = could not find container \"84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5\": container with ID starting with 84906ee6a197c3e10a0cfe0d460c766c8e4b15d699b2689539fdc5e789678db5 not found: ID does not exist" Jul 14 18:18:47.704074 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.704054 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h45rf"] Jul 14 18:18:47.710704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.710684 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h45rf"] Jul 14 18:18:47.724031 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.724012 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b54dfbf98-z9vft"] Jul 14 18:18:47.731311 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:47.731291 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b54dfbf98-z9vft"] Jul 14 18:18:48.692663 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:48.692625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjfht" event={"ID":"4acecd60-5f69-4332-9b0a-2597b299f60c","Type":"ContainerStarted","Data":"212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a"} Jul 14 18:18:48.692663 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:48.692659 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-sjfht" podUID="4acecd60-5f69-4332-9b0a-2597b299f60c" containerName="authorino" containerID="cri-o://212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a" gracePeriod=30 Jul 14 18:18:48.714445 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:48.714392 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-sjfht" podStartSLOduration=1.96214197 podStartE2EDuration="4.714374693s" totalImagesPullingTime="2.752232723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:48.71366633 +0000 UTC m=+528.145521766" watchObservedRunningTime="2026-07-14 18:18:48.714374693 +0000 UTC m=+528.146230162" Jul 14 18:18:48.984109 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:48.984079 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjfht" Jul 14 18:18:49.117424 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.117376 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxr7b\" (UniqueName: \"kubernetes.io/projected/4acecd60-5f69-4332-9b0a-2597b299f60c-kube-api-access-qxr7b\") pod \"4acecd60-5f69-4332-9b0a-2597b299f60c\" (UID: \"4acecd60-5f69-4332-9b0a-2597b299f60c\") " Jul 14 18:18:49.119424 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.119392 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acecd60-5f69-4332-9b0a-2597b299f60c-kube-api-access-qxr7b" (OuterVolumeSpecName: "kube-api-access-qxr7b") pod "4acecd60-5f69-4332-9b0a-2597b299f60c" (UID: "4acecd60-5f69-4332-9b0a-2597b299f60c"). InnerVolumeSpecName "kube-api-access-qxr7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:18:49.207882 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.207858 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04d06fbd-38a3-4e3c-af7d-1f9767e8c04f" path="/var/lib/kubelet/pods/04d06fbd-38a3-4e3c-af7d-1f9767e8c04f/volumes" Jul 14 18:18:49.208176 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.208164 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3" path="/var/lib/kubelet/pods/4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3/volumes" Jul 14 18:18:49.218057 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.218034 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxr7b\" (UniqueName: \"kubernetes.io/projected/4acecd60-5f69-4332-9b0a-2597b299f60c-kube-api-access-qxr7b\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:18:49.700286 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.700195 2579 generic.go:356] "Generic (PLEG): container finished" podID="4acecd60-5f69-4332-9b0a-2597b299f60c" containerID="212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a" exitCode=0 Jul 14 18:18:49.700286 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.700235 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sjfht" Jul 14 18:18:49.700286 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.700276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjfht" event={"ID":"4acecd60-5f69-4332-9b0a-2597b299f60c","Type":"ContainerDied","Data":"212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a"} Jul 14 18:18:49.700813 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.700311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sjfht" event={"ID":"4acecd60-5f69-4332-9b0a-2597b299f60c","Type":"ContainerDied","Data":"d07fccba5265037a244e33398f1b0f45d541e8715da3ae5135ee81a08d3dc1c1"} Jul 14 18:18:49.700813 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.700328 2579 scope.go:117] "RemoveContainer" containerID="212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a" Jul 14 18:18:49.708853 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.708831 2579 scope.go:117] "RemoveContainer" containerID="212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a" Jul 14 18:18:49.709126 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:18:49.709107 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a\": container with ID starting with 212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a not found: ID does not exist" containerID="212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a" Jul 14 18:18:49.709198 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.709138 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a"} err="failed to get container status \"212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a\": rpc error: code = NotFound desc = could not find container \"212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a\": container with ID starting with 212448a8d514d5269ba61dbd54115c36456896cea4c429da65f264518714007a not found: ID does not exist" Jul 14 18:18:49.720081 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.720054 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjfht"] Jul 14 18:18:49.728127 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:49.728105 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sjfht"] Jul 14 18:18:51.206930 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:18:51.206898 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4acecd60-5f69-4332-9b0a-2597b299f60c" path="/var/lib/kubelet/pods/4acecd60-5f69-4332-9b0a-2597b299f60c/volumes" Jul 14 18:19:57.065661 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.065624 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7"] Jul 14 18:19:57.066110 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066002 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3" containerName="authorino" Jul 14 18:19:57.066110 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066014 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3" containerName="authorino" Jul 14 18:19:57.066110 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066029 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4acecd60-5f69-4332-9b0a-2597b299f60c" containerName="authorino" Jul 14 18:19:57.066110 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066034 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4acecd60-5f69-4332-9b0a-2597b299f60c" containerName="authorino" Jul 14 18:19:57.066110 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066045 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d06fbd-38a3-4e3c-af7d-1f9767e8c04f" containerName="authorino" Jul 14 18:19:57.066110 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066050 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d06fbd-38a3-4e3c-af7d-1f9767e8c04f" containerName="authorino" Jul 14 18:19:57.066313 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066129 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4acecd60-5f69-4332-9b0a-2597b299f60c" containerName="authorino" Jul 14 18:19:57.066313 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066139 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4a4fa4-a245-4a10-bfee-b2ba86bdd5b3" containerName="authorino" Jul 14 18:19:57.066313 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.066145 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d06fbd-38a3-4e3c-af7d-1f9767e8c04f" containerName="authorino" Jul 14 18:19:57.070832 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.070792 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.073620 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.073592 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6s2bs\"" Jul 14 18:19:57.074694 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.074671 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 18:19:57.074808 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.074675 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 14 18:19:57.074808 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.074710 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 18:19:57.085926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.085902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7"] Jul 14 18:19:57.130614 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.130536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.130801 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.130618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93aaa300-e330-45e3-a79d-a30df5e60540-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.130801 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.130646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rm4d\" (UniqueName: \"kubernetes.io/projected/93aaa300-e330-45e3-a79d-a30df5e60540-kube-api-access-6rm4d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.130801 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.130682 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.130801 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.130708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.130801 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.130773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.231903 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.231870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.231903 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.231907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.232170 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.232128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.232279 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.232258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.232364 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.232297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93aaa300-e330-45e3-a79d-a30df5e60540-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.232364 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.232344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.232441 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.232424 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.232476 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.232325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rm4d\" (UniqueName: \"kubernetes.io/projected/93aaa300-e330-45e3-a79d-a30df5e60540-kube-api-access-6rm4d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.232661 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.232644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.234198 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.234175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93aaa300-e330-45e3-a79d-a30df5e60540-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.234526 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.234509 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93aaa300-e330-45e3-a79d-a30df5e60540-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.242853 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.242827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rm4d\" (UniqueName: \"kubernetes.io/projected/93aaa300-e330-45e3-a79d-a30df5e60540-kube-api-access-6rm4d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pcqj7\" (UID: \"93aaa300-e330-45e3-a79d-a30df5e60540\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.381583 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.381464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:19:57.520785 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:57.520759 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7"] Jul 14 18:19:57.522337 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:19:57.522315 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93aaa300_e330_45e3_a79d_a30df5e60540.slice/crio-844bed4ab373f219bd7a753f1b436d71ad44f2c4f9605c1e74f45052a7ad7127 WatchSource:0}: Error finding container 844bed4ab373f219bd7a753f1b436d71ad44f2c4f9605c1e74f45052a7ad7127: Status 404 returned error can't find the container with id 844bed4ab373f219bd7a753f1b436d71ad44f2c4f9605c1e74f45052a7ad7127 Jul 14 18:19:58.047465 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:58.047426 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" event={"ID":"93aaa300-e330-45e3-a79d-a30df5e60540","Type":"ContainerStarted","Data":"844bed4ab373f219bd7a753f1b436d71ad44f2c4f9605c1e74f45052a7ad7127"} Jul 14 18:19:59.053998 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:59.053961 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" event={"ID":"93aaa300-e330-45e3-a79d-a30df5e60540","Type":"ContainerStarted","Data":"235ffcbca1c16cd9cf6fd1494d8ad0e6841127b7bf27f441bfc827a19c5b2276"} Jul 14 18:19:59.076474 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:19:59.076419 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" podStartSLOduration=0.9452404 podStartE2EDuration="2.076403025s" totalImagesPullingTime="1.131162625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:19:59.074889743 +0000 UTC m=+598.506745180" watchObservedRunningTime="2026-07-14 18:19:59.076403025 +0000 UTC m=+598.508258458" Jul 14 18:20:01.052859 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:01.052832 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:20:01.053312 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:01.053299 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:20:07.382204 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:07.382168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:20:07.383538 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:07.382226 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:20:07.394806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:07.394783 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:20:08.109143 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:08.109108 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pcqj7" Jul 14 18:20:21.891440 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:21.891401 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws"] Jul 14 18:20:21.898179 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:21.898156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:21.901054 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:21.901029 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 14 18:20:21.916370 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:21.916341 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws"] Jul 14 18:20:22.059132 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.059091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.059132 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.059141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.059347 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.059175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgxwt\" (UniqueName: \"kubernetes.io/projected/207dc962-216f-4514-b67e-84fb2ea9d460-kube-api-access-hgxwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.059347 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.059230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.059347 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.059275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207dc962-216f-4514-b67e-84fb2ea9d460-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.059455 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.059361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.160770 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.160676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.160770 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.160725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.160770 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.160758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.161067 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.160808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgxwt\" (UniqueName: \"kubernetes.io/projected/207dc962-216f-4514-b67e-84fb2ea9d460-kube-api-access-hgxwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.161067 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.160847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.161067 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.160887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207dc962-216f-4514-b67e-84fb2ea9d460-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.161245 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.161128 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.161245 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.161234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.161330 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.161279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.163201 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.163182 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207dc962-216f-4514-b67e-84fb2ea9d460-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.163323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.163306 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207dc962-216f-4514-b67e-84fb2ea9d460-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.172098 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.172064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgxwt\" (UniqueName: \"kubernetes.io/projected/207dc962-216f-4514-b67e-84fb2ea9d460-kube-api-access-hgxwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zb2ws\" (UID: \"207dc962-216f-4514-b67e-84fb2ea9d460\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.207661 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.207634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:22.353222 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.353195 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws"] Jul 14 18:20:22.355248 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:20:22.355222 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207dc962_216f_4514_b67e_84fb2ea9d460.slice/crio-f457d713b8b82d9d17444359e856a128c60748dbaa633cc3d4171d7532235b2c WatchSource:0}: Error finding container f457d713b8b82d9d17444359e856a128c60748dbaa633cc3d4171d7532235b2c: Status 404 returned error can't find the container with id f457d713b8b82d9d17444359e856a128c60748dbaa633cc3d4171d7532235b2c Jul 14 18:20:22.356636 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:22.356621 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:20:23.176299 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:23.176266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" event={"ID":"207dc962-216f-4514-b67e-84fb2ea9d460","Type":"ContainerStarted","Data":"18fcf2862388595be25022d29b0a08eff52940334a97ec401784576264a02d8e"} Jul 14 18:20:23.176866 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:23.176331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" event={"ID":"207dc962-216f-4514-b67e-84fb2ea9d460","Type":"ContainerStarted","Data":"f457d713b8b82d9d17444359e856a128c60748dbaa633cc3d4171d7532235b2c"} Jul 14 18:20:23.197515 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:23.197426 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" podStartSLOduration=1.9718786609999999 podStartE2EDuration="2.197410886s" totalImagesPullingTime="225.532225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:20:23.195281563 +0000 UTC m=+622.627137006" watchObservedRunningTime="2026-07-14 18:20:23.197410886 +0000 UTC m=+622.629266298" Jul 14 18:20:25.670416 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.670377 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r"] Jul 14 18:20:25.674205 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.674182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.676903 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.676882 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 14 18:20:25.698607 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.698567 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r"] Jul 14 18:20:25.788928 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.788895 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnb5p\" (UniqueName: \"kubernetes.io/projected/816e572a-53b3-471f-bdc1-597c402a9804-kube-api-access-xnb5p\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.789099 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.788970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.789099 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.788994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.789099 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.789026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816e572a-53b3-471f-bdc1-597c402a9804-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.789099 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.789074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.789250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.789106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890282 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnb5p\" (UniqueName: \"kubernetes.io/projected/816e572a-53b3-471f-bdc1-597c402a9804-kube-api-access-xnb5p\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890474 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890474 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890474 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816e572a-53b3-471f-bdc1-597c402a9804-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890474 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890740 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890840 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890970 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.890970 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.890928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.892789 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.892764 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816e572a-53b3-471f-bdc1-597c402a9804-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.893065 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.893042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816e572a-53b3-471f-bdc1-597c402a9804-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.904829 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.904802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnb5p\" (UniqueName: \"kubernetes.io/projected/816e572a-53b3-471f-bdc1-597c402a9804-kube-api-access-xnb5p\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r\" (UID: \"816e572a-53b3-471f-bdc1-597c402a9804\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:25.983708 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:25.983677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:26.119809 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:26.119782 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r"] Jul 14 18:20:26.121200 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:20:26.121173 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod816e572a_53b3_471f_bdc1_597c402a9804.slice/crio-9fbe342c6dc3a55413d8706e24069cebc2ae543caf19de53fb5ed7ceaf69dbb6 WatchSource:0}: Error finding container 9fbe342c6dc3a55413d8706e24069cebc2ae543caf19de53fb5ed7ceaf69dbb6: Status 404 returned error can't find the container with id 9fbe342c6dc3a55413d8706e24069cebc2ae543caf19de53fb5ed7ceaf69dbb6 Jul 14 18:20:26.194195 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:26.194162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" event={"ID":"816e572a-53b3-471f-bdc1-597c402a9804","Type":"ContainerStarted","Data":"9fbe342c6dc3a55413d8706e24069cebc2ae543caf19de53fb5ed7ceaf69dbb6"} Jul 14 18:20:27.206217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:27.206173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" event={"ID":"816e572a-53b3-471f-bdc1-597c402a9804","Type":"ContainerStarted","Data":"58666239966cf27e4b65ae24f9723be5611c8bfb984bb6de61998b25c0b04e85"} Jul 14 18:20:27.230148 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:27.230052 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" podStartSLOduration=2.054132973 podStartE2EDuration="2.230032643s" totalImagesPullingTime="175.89967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:20:27.229687143 +0000 UTC m=+626.661542577" watchObservedRunningTime="2026-07-14 18:20:27.230032643 +0000 UTC m=+626.661888080" Jul 14 18:20:32.208656 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:32.208610 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:32.209122 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:32.208704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:32.221303 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:32.221272 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:32.239192 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:32.239161 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zb2ws" Jul 14 18:20:35.984827 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:35.984785 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:35.985277 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:35.985082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:35.997711 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:35.997674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:20:36.262416 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:20:36.262336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r" Jul 14 18:21:36.900506 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:36.900471 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c9c594c5-v45s2"] Jul 14 18:21:36.903877 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:36.903858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9c594c5-v45s2" Jul 14 18:21:36.921800 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:36.921771 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c9c594c5-v45s2"] Jul 14 18:21:37.008237 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.008201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc-tls-cert\") pod \"authorino-6c9c594c5-v45s2\" (UID: \"effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc\") " pod="kuadrant-system/authorino-6c9c594c5-v45s2" Jul 14 18:21:37.008404 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.008252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jrhj\" (UniqueName: \"kubernetes.io/projected/effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc-kube-api-access-7jrhj\") pod \"authorino-6c9c594c5-v45s2\" (UID: \"effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc\") " pod="kuadrant-system/authorino-6c9c594c5-v45s2" Jul 14 18:21:37.108931 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.108889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc-tls-cert\") pod \"authorino-6c9c594c5-v45s2\" (UID: \"effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc\") " pod="kuadrant-system/authorino-6c9c594c5-v45s2" Jul 14 18:21:37.109101 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.108941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jrhj\" (UniqueName: \"kubernetes.io/projected/effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc-kube-api-access-7jrhj\") pod \"authorino-6c9c594c5-v45s2\" (UID: \"effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc\") " pod="kuadrant-system/authorino-6c9c594c5-v45s2" Jul 14 18:21:37.111494 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.111462 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc-tls-cert\") pod \"authorino-6c9c594c5-v45s2\" (UID: \"effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc\") " pod="kuadrant-system/authorino-6c9c594c5-v45s2" Jul 14 18:21:37.120337 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.120313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jrhj\" (UniqueName: \"kubernetes.io/projected/effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc-kube-api-access-7jrhj\") pod \"authorino-6c9c594c5-v45s2\" (UID: \"effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc\") " pod="kuadrant-system/authorino-6c9c594c5-v45s2" Jul 14 18:21:37.212584 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.212536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9c594c5-v45s2" Jul 14 18:21:37.383235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.383202 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c9c594c5-v45s2"] Jul 14 18:21:37.394402 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:21:37.394374 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeffa659c_c9eb_4e9f_8bf5_2d12d1bcabcc.slice/crio-1472606f91b27e5c8b3bb048e46b88da6fc226407f44b276cbd65151969ee0d0 WatchSource:0}: Error finding container 1472606f91b27e5c8b3bb048e46b88da6fc226407f44b276cbd65151969ee0d0: Status 404 returned error can't find the container with id 1472606f91b27e5c8b3bb048e46b88da6fc226407f44b276cbd65151969ee0d0 Jul 14 18:21:37.568160 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:37.568065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9c594c5-v45s2" event={"ID":"effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc","Type":"ContainerStarted","Data":"1472606f91b27e5c8b3bb048e46b88da6fc226407f44b276cbd65151969ee0d0"} Jul 14 18:21:38.578697 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:38.578650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9c594c5-v45s2" event={"ID":"effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc","Type":"ContainerStarted","Data":"d3c96c48e42c0d9ee732b8c3c260ba829f6265343925c1e3977dec71fbe07ebf"} Jul 14 18:21:38.606926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:38.606859 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c9c594c5-v45s2" podStartSLOduration=2.170825898 podStartE2EDuration="2.606820163s" totalImagesPullingTime="435.994265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:21:38.606607434 +0000 UTC m=+698.038462866" watchObservedRunningTime="2026-07-14 18:21:38.606820163 +0000 UTC m=+698.038675597" Jul 14 18:21:38.651255 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:38.651216 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67b9fc9f46-zkrqg"] Jul 14 18:21:38.651435 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:38.651404 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" podUID="a7926c46-9672-40aa-86ee-b51f8876492c" containerName="authorino" containerID="cri-o://351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db" gracePeriod=30 Jul 14 18:21:38.903179 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:38.903152 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:21:39.027669 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.027630 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7926c46-9672-40aa-86ee-b51f8876492c-tls-cert\") pod \"a7926c46-9672-40aa-86ee-b51f8876492c\" (UID: \"a7926c46-9672-40aa-86ee-b51f8876492c\") " Jul 14 18:21:39.027852 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.027727 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49jqq\" (UniqueName: \"kubernetes.io/projected/a7926c46-9672-40aa-86ee-b51f8876492c-kube-api-access-49jqq\") pod \"a7926c46-9672-40aa-86ee-b51f8876492c\" (UID: \"a7926c46-9672-40aa-86ee-b51f8876492c\") " Jul 14 18:21:39.029756 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.029724 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7926c46-9672-40aa-86ee-b51f8876492c-kube-api-access-49jqq" (OuterVolumeSpecName: "kube-api-access-49jqq") pod "a7926c46-9672-40aa-86ee-b51f8876492c" (UID: "a7926c46-9672-40aa-86ee-b51f8876492c"). InnerVolumeSpecName "kube-api-access-49jqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:21:39.038336 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.038307 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7926c46-9672-40aa-86ee-b51f8876492c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a7926c46-9672-40aa-86ee-b51f8876492c" (UID: "a7926c46-9672-40aa-86ee-b51f8876492c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:21:39.129128 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.129097 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7926c46-9672-40aa-86ee-b51f8876492c-tls-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:21:39.129128 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.129127 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jqq\" (UniqueName: \"kubernetes.io/projected/a7926c46-9672-40aa-86ee-b51f8876492c-kube-api-access-49jqq\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:21:39.586570 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.586454 2579 generic.go:356] "Generic (PLEG): container finished" podID="a7926c46-9672-40aa-86ee-b51f8876492c" containerID="351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db" exitCode=0 Jul 14 18:21:39.586570 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.586497 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" Jul 14 18:21:39.587046 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.586544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" event={"ID":"a7926c46-9672-40aa-86ee-b51f8876492c","Type":"ContainerDied","Data":"351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db"} Jul 14 18:21:39.587046 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.586614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67b9fc9f46-zkrqg" event={"ID":"a7926c46-9672-40aa-86ee-b51f8876492c","Type":"ContainerDied","Data":"af26c4ed09108f3e70d04d02f1b4903816d182c47d9376475f0f73861ec21e8e"} Jul 14 18:21:39.587046 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.586633 2579 scope.go:117] "RemoveContainer" containerID="351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db" Jul 14 18:21:39.597422 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.597405 2579 scope.go:117] "RemoveContainer" containerID="351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db" Jul 14 18:21:39.597747 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:21:39.597727 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db\": container with ID starting with 351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db not found: ID does not exist" containerID="351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db" Jul 14 18:21:39.597823 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.597755 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db"} err="failed to get container status \"351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db\": rpc error: code = NotFound desc = could not find container \"351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db\": container with ID starting with 351af9a26a661517a21c626e5cab7d1bd6ef859bfe9ecf27224f81d3c2a8e4db not found: ID does not exist" Jul 14 18:21:39.608377 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.608350 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67b9fc9f46-zkrqg"] Jul 14 18:21:39.615706 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:39.615680 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-67b9fc9f46-zkrqg"] Jul 14 18:21:41.206707 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:21:41.206667 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7926c46-9672-40aa-86ee-b51f8876492c" path="/var/lib/kubelet/pods/a7926c46-9672-40aa-86ee-b51f8876492c/volumes" Jul 14 18:25:01.083309 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:25:01.083280 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:25:01.084880 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:25:01.084859 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:27:48.072632 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:27:48.072597 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb66dd77f-5r68z"] Jul 14 18:28:13.092665 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.092622 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bb66dd77f-5r68z" podUID="91363686-f027-461e-a47a-c1f5583d9d3d" containerName="console" containerID="cri-o://b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9" gracePeriod=15 Jul 14 18:28:13.350054 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.349991 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:28:13.443335 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443298 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-oauth-config\") pod \"91363686-f027-461e-a47a-c1f5583d9d3d\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " Jul 14 18:28:13.443520 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443383 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjpz2\" (UniqueName: \"kubernetes.io/projected/91363686-f027-461e-a47a-c1f5583d9d3d-kube-api-access-gjpz2\") pod \"91363686-f027-461e-a47a-c1f5583d9d3d\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " Jul 14 18:28:13.443520 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443406 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-trusted-ca-bundle\") pod \"91363686-f027-461e-a47a-c1f5583d9d3d\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " Jul 14 18:28:13.443520 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443425 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-console-config\") pod \"91363686-f027-461e-a47a-c1f5583d9d3d\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " Jul 14 18:28:13.443520 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443443 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-oauth-serving-cert\") pod \"91363686-f027-461e-a47a-c1f5583d9d3d\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " Jul 14 18:28:13.443520 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443485 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-service-ca\") pod \"91363686-f027-461e-a47a-c1f5583d9d3d\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " Jul 14 18:28:13.443880 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443528 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-serving-cert\") pod \"91363686-f027-461e-a47a-c1f5583d9d3d\" (UID: \"91363686-f027-461e-a47a-c1f5583d9d3d\") " Jul 14 18:28:13.443950 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443887 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-console-config" (OuterVolumeSpecName: "console-config") pod "91363686-f027-461e-a47a-c1f5583d9d3d" (UID: "91363686-f027-461e-a47a-c1f5583d9d3d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:28:13.444006 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443937 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "91363686-f027-461e-a47a-c1f5583d9d3d" (UID: "91363686-f027-461e-a47a-c1f5583d9d3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:28:13.444006 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.443958 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-service-ca" (OuterVolumeSpecName: "service-ca") pod "91363686-f027-461e-a47a-c1f5583d9d3d" (UID: "91363686-f027-461e-a47a-c1f5583d9d3d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:28:13.444298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.444270 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "91363686-f027-461e-a47a-c1f5583d9d3d" (UID: "91363686-f027-461e-a47a-c1f5583d9d3d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:28:13.445790 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.445765 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "91363686-f027-461e-a47a-c1f5583d9d3d" (UID: "91363686-f027-461e-a47a-c1f5583d9d3d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:28:13.445884 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.445843 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91363686-f027-461e-a47a-c1f5583d9d3d-kube-api-access-gjpz2" (OuterVolumeSpecName: "kube-api-access-gjpz2") pod "91363686-f027-461e-a47a-c1f5583d9d3d" (UID: "91363686-f027-461e-a47a-c1f5583d9d3d"). InnerVolumeSpecName "kube-api-access-gjpz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:28:13.446029 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.446011 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "91363686-f027-461e-a47a-c1f5583d9d3d" (UID: "91363686-f027-461e-a47a-c1f5583d9d3d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:28:13.544605 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.544573 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-serving-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:28:13.544605 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.544599 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91363686-f027-461e-a47a-c1f5583d9d3d-console-oauth-config\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:28:13.544605 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.544609 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjpz2\" (UniqueName: \"kubernetes.io/projected/91363686-f027-461e-a47a-c1f5583d9d3d-kube-api-access-gjpz2\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:28:13.544605 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.544617 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-trusted-ca-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:28:13.544891 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.544626 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-console-config\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:28:13.544891 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.544634 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-oauth-serving-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:28:13.544891 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.544643 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91363686-f027-461e-a47a-c1f5583d9d3d-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:28:13.659987 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.659893 2579 generic.go:356] "Generic (PLEG): container finished" podID="91363686-f027-461e-a47a-c1f5583d9d3d" containerID="b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9" exitCode=0 Jul 14 18:28:13.659987 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.659955 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb66dd77f-5r68z" Jul 14 18:28:13.660170 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.659980 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb66dd77f-5r68z" event={"ID":"91363686-f027-461e-a47a-c1f5583d9d3d","Type":"ContainerDied","Data":"b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9"} Jul 14 18:28:13.660170 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.660060 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb66dd77f-5r68z" event={"ID":"91363686-f027-461e-a47a-c1f5583d9d3d","Type":"ContainerDied","Data":"d289ad4a146bdd56a20cea31d679153ae6ec5dbfbe624c6d746b5642bfc9392f"} Jul 14 18:28:13.660170 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.660083 2579 scope.go:117] "RemoveContainer" containerID="b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9" Jul 14 18:28:13.669503 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.669483 2579 scope.go:117] "RemoveContainer" containerID="b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9" Jul 14 18:28:13.669858 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:28:13.669838 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9\": container with ID starting with b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9 not found: ID does not exist" containerID="b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9" Jul 14 18:28:13.669918 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.669868 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9"} err="failed to get container status \"b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9\": rpc error: code = NotFound desc = could not find container \"b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9\": container with ID starting with b793554fca153386dea7c6a6eddcd0d15bdb0cb234be67c151b83ba3413722d9 not found: ID does not exist" Jul 14 18:28:13.690308 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.690276 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb66dd77f-5r68z"] Jul 14 18:28:13.694409 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:13.694384 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bb66dd77f-5r68z"] Jul 14 18:28:15.207127 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:28:15.207089 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91363686-f027-461e-a47a-c1f5583d9d3d" path="/var/lib/kubelet/pods/91363686-f027-461e-a47a-c1f5583d9d3d/volumes" Jul 14 18:30:01.119688 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:30:01.119662 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:30:01.122315 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:30:01.122293 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:35:01.150661 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:01.150629 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:35:01.155112 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:01.155084 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:35:40.151702 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.151625 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9"] Jul 14 18:35:40.152134 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.151963 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91363686-f027-461e-a47a-c1f5583d9d3d" containerName="console" Jul 14 18:35:40.152134 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.151973 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91363686-f027-461e-a47a-c1f5583d9d3d" containerName="console" Jul 14 18:35:40.152134 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.151983 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7926c46-9672-40aa-86ee-b51f8876492c" containerName="authorino" Jul 14 18:35:40.152134 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.151989 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7926c46-9672-40aa-86ee-b51f8876492c" containerName="authorino" Jul 14 18:35:40.152134 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.152037 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91363686-f027-461e-a47a-c1f5583d9d3d" containerName="console" Jul 14 18:35:40.152134 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.152047 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7926c46-9672-40aa-86ee-b51f8876492c" containerName="authorino" Jul 14 18:35:40.154911 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.154892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.157442 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.157421 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-499cb51a-gw-openshift-default-dockercfg-vjllp\"" Jul 14 18:35:40.168148 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.168124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9"] Jul 14 18:35:40.272087 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272056 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-envoy\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.272087 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-token\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.272087 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istiod-ca-cert\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.272378 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-certs\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.272378 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-socket\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.272378 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-credential-socket\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.272378 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-data\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.272378 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkjvv\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-kube-api-access-hkjvv\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.272543 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.272385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-podinfo\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.372974 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.372935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-envoy\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.372974 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.372975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-token\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.372991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istiod-ca-cert\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-certs\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-socket\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-credential-socket\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-data\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkjvv\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-kube-api-access-hkjvv\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-podinfo\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373627 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373593 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-credential-socket\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373769 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-data\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373769 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373657 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-certs\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.373769 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.373708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-socket\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.374140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.374115 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istiod-ca-cert\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.375477 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.375458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-envoy\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.375712 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.375694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-podinfo\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.383372 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.383329 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-token\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.383505 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.383490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkjvv\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-kube-api-access-hkjvv\") pod \"e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.465419 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.465384 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:40.608998 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.608971 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9"] Jul 14 18:35:40.611611 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:35:40.611578 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9567f2_4aa5_4213_8ef8_1666dd0de6ab.slice/crio-db0a933ad167fa5073aba035e4f984ba63affca6d1140cdf5af52342bc451081 WatchSource:0}: Error finding container db0a933ad167fa5073aba035e4f984ba63affca6d1140cdf5af52342bc451081: Status 404 returned error can't find the container with id db0a933ad167fa5073aba035e4f984ba63affca6d1140cdf5af52342bc451081 Jul 14 18:35:40.613755 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.613724 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:35:40.613851 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.613789 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:35:40.613851 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.613821 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:35:40.949783 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.949677 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" event={"ID":"af9567f2-4aa5-4213-8ef8-1666dd0de6ab","Type":"ContainerStarted","Data":"b2ca51d472c8c45a5aa9dfd6402954c75722e8c7b039905777c1965e4a5069c1"} Jul 14 18:35:40.949783 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.949733 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" event={"ID":"af9567f2-4aa5-4213-8ef8-1666dd0de6ab","Type":"ContainerStarted","Data":"db0a933ad167fa5073aba035e4f984ba63affca6d1140cdf5af52342bc451081"} Jul 14 18:35:40.972935 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:40.972874 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" podStartSLOduration=0.972851314 podStartE2EDuration="972.851314ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:35:40.97159637 +0000 UTC m=+1540.403451803" watchObservedRunningTime="2026-07-14 18:35:40.972851314 +0000 UTC m=+1540.404706752" Jul 14 18:35:41.465601 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:41.465536 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:41.470501 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:41.470474 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:41.955647 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:41.955564 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:41.956674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:41.956652 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:56.957583 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:56.957532 2579 patch_prober.go:28] interesting pod/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 14 18:35:56.957998 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:56.957610 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 14 18:35:57.033909 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:57.033877 2579 generic.go:356] "Generic (PLEG): container finished" podID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerID="b2ca51d472c8c45a5aa9dfd6402954c75722e8c7b039905777c1965e4a5069c1" exitCode=0 Jul 14 18:35:57.034069 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:57.033952 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" event={"ID":"af9567f2-4aa5-4213-8ef8-1666dd0de6ab","Type":"ContainerDied","Data":"b2ca51d472c8c45a5aa9dfd6402954c75722e8c7b039905777c1965e4a5069c1"} Jul 14 18:35:57.034436 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:57.034417 2579 scope.go:117] "RemoveContainer" containerID="b2ca51d472c8c45a5aa9dfd6402954c75722e8c7b039905777c1965e4a5069c1" Jul 14 18:35:57.035389 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:57.035363 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:35:57.035486 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:57.035442 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:35:57.035547 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:57.035489 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:35:58.044437 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:58.044390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" event={"ID":"af9567f2-4aa5-4213-8ef8-1666dd0de6ab","Type":"ContainerStarted","Data":"1e952ce233a93031911aad89fcae6d881ab25b77949506ff7b5bdb93aa84844e"} Jul 14 18:35:58.465738 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:58.465705 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:35:58.465970 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:58.465944 2579 patch_prober.go:28] interesting pod/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 14 18:35:58.466050 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:58.466027 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 14 18:35:59.466404 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:59.466368 2579 patch_prober.go:28] interesting pod/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 14 18:35:59.466836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:35:59.466429 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 14 18:36:00.466064 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:00.466025 2579 patch_prober.go:28] interesting pod/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 14 18:36:00.466275 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:00.466091 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 14 18:36:01.465874 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:01.465828 2579 patch_prober.go:28] interesting pod/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 14 18:36:01.466275 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:01.465910 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 14 18:36:02.469800 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:02.469768 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:36:02.470199 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:02.470031 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:36:02.470757 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:02.470739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:36:46.105704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:46.105666 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9"] Jul 14 18:36:46.106135 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:46.105896 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" containerID="cri-o://1e952ce233a93031911aad89fcae6d881ab25b77949506ff7b5bdb93aa84844e" gracePeriod=30 Jul 14 18:36:47.472480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:47.472442 2579 patch_prober.go:28] interesting pod/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:36:47.472874 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:47.472515 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:36:51.323359 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.323319 2579 generic.go:356] "Generic (PLEG): container finished" podID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerID="1e952ce233a93031911aad89fcae6d881ab25b77949506ff7b5bdb93aa84844e" exitCode=0 Jul 14 18:36:51.323776 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.323396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" event={"ID":"af9567f2-4aa5-4213-8ef8-1666dd0de6ab","Type":"ContainerDied","Data":"1e952ce233a93031911aad89fcae6d881ab25b77949506ff7b5bdb93aa84844e"} Jul 14 18:36:51.323776 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.323463 2579 scope.go:117] "RemoveContainer" containerID="b2ca51d472c8c45a5aa9dfd6402954c75722e8c7b039905777c1965e4a5069c1" Jul 14 18:36:51.369465 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.369404 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:36:51.420654 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420623 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-envoy\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.420817 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420691 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-podinfo\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.420817 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-socket\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.420817 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420793 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-token\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.421072 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-data\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.421072 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420846 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkjvv\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-kube-api-access-hkjvv\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.421072 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420873 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istiod-ca-cert\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.421072 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420898 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-certs\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.421072 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.420923 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-credential-socket\") pod \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\" (UID: \"af9567f2-4aa5-4213-8ef8-1666dd0de6ab\") " Jul 14 18:36:51.421575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.421360 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:36:51.421575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.421387 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:36:51.421575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.421506 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:36:51.421575 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.421563 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:36:51.421862 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.421838 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:36:51.423781 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.423756 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:36:51.423900 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.423827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:36:51.423900 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.423828 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-kube-api-access-hkjvv" (OuterVolumeSpecName: "kube-api-access-hkjvv") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "kube-api-access-hkjvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:36:51.424101 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.424076 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af9567f2-4aa5-4213-8ef8-1666dd0de6ab" (UID: "af9567f2-4aa5-4213-8ef8-1666dd0de6ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:36:51.521899 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521867 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:51.521899 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521897 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:51.521899 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521907 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:51.522140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521915 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkjvv\" (UniqueName: \"kubernetes.io/projected/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-kube-api-access-hkjvv\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:51.522140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521925 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:51.522140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521934 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:51.522140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521942 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:51.522140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521949 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:51.522140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:51.521957 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9567f2-4aa5-4213-8ef8-1666dd0de6ab-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:36:52.330461 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:52.330434 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" Jul 14 18:36:52.331018 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:52.330475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9" event={"ID":"af9567f2-4aa5-4213-8ef8-1666dd0de6ab","Type":"ContainerDied","Data":"db0a933ad167fa5073aba035e4f984ba63affca6d1140cdf5af52342bc451081"} Jul 14 18:36:52.331018 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:52.330534 2579 scope.go:117] "RemoveContainer" containerID="1e952ce233a93031911aad89fcae6d881ab25b77949506ff7b5bdb93aa84844e" Jul 14 18:36:52.354222 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:52.354194 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9"] Jul 14 18:36:52.359079 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:52.359058 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-499cb51a-gw-openshift-default-89f7f8df9-6dtz9"] Jul 14 18:36:53.206968 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:36:53.206884 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" path="/var/lib/kubelet/pods/af9567f2-4aa5-4213-8ef8-1666dd0de6ab/volumes" Jul 14 18:37:26.051125 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.051094 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv"] Jul 14 18:37:26.051580 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.051416 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" Jul 14 18:37:26.051580 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.051426 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" Jul 14 18:37:26.051580 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.051439 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" Jul 14 18:37:26.051580 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.051445 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" Jul 14 18:37:26.051580 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.051505 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" Jul 14 18:37:26.051580 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.051513 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9567f2-4aa5-4213-8ef8-1666dd0de6ab" containerName="istio-proxy" Jul 14 18:37:26.054744 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.054709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.057519 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.057489 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e2e80b9d-openshift-default-dockercfg-vnp95\"" Jul 14 18:37:26.070324 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.070302 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv"] Jul 14 18:37:26.108873 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.108839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-certs\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.108873 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.108879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-socket\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.109137 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.108899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-credential-socket\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.109137 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.109000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-podinfo\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.109137 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.109054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-token\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.109137 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.109086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9lsc\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-kube-api-access-z9lsc\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.109288 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.109175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-envoy\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.109288 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.109201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-data\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.109288 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.109233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istiod-ca-cert\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210194 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-podinfo\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210194 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-token\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210467 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9lsc\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-kube-api-access-z9lsc\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210467 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-envoy\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210467 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-data\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210467 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istiod-ca-cert\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210467 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-certs\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210467 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-socket\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210467 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-credential-socket\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210887 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-credential-socket\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210955 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-data\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.210955 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.210936 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-certs\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.211038 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.211018 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-socket\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.211143 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.211125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istiod-ca-cert\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.213087 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.213067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-envoy\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.213399 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.213379 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-podinfo\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.219589 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.219545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-token\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.219999 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.219978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9lsc\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-kube-api-access-z9lsc\") pod \"e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.366447 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.366354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:26.508001 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.507968 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv"] Jul 14 18:37:26.508315 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:37:26.508282 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2ea88a_8d94_4d82_9988_debfd7a1dc22.slice/crio-565475a94e42be377fbb3541b6d7a07a8464ef3eca8111e7502ca75d98481410 WatchSource:0}: Error finding container 565475a94e42be377fbb3541b6d7a07a8464ef3eca8111e7502ca75d98481410: Status 404 returned error can't find the container with id 565475a94e42be377fbb3541b6d7a07a8464ef3eca8111e7502ca75d98481410 Jul 14 18:37:26.510758 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.510709 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:37:26.510849 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.510807 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:37:26.510896 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:26.510858 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:37:27.515067 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:27.515035 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" event={"ID":"4b2ea88a-8d94-4d82-9988-debfd7a1dc22","Type":"ContainerStarted","Data":"988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75"} Jul 14 18:37:27.515470 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:27.515089 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" event={"ID":"4b2ea88a-8d94-4d82-9988-debfd7a1dc22","Type":"ContainerStarted","Data":"565475a94e42be377fbb3541b6d7a07a8464ef3eca8111e7502ca75d98481410"} Jul 14 18:37:27.561428 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:27.561358 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" podStartSLOduration=1.561339331 podStartE2EDuration="1.561339331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:37:27.55694418 +0000 UTC m=+1646.988799606" watchObservedRunningTime="2026-07-14 18:37:27.561339331 +0000 UTC m=+1646.993194765" Jul 14 18:37:28.367567 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:28.367526 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:28.373268 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:28.373239 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:28.520313 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:28.520283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:28.521340 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:28.521320 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:43.655019 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:43.653758 2579 patch_prober.go:28] interesting pod/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:45704->10.133.0.50:15021: read: connection reset by peer" start-of-body= Jul 14 18:37:43.655019 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:43.653852 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:45704->10.133.0.50:15021: read: connection reset by peer" Jul 14 18:37:44.605109 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:44.605077 2579 generic.go:356] "Generic (PLEG): container finished" podID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerID="988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75" exitCode=0 Jul 14 18:37:44.605323 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:44.605154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" event={"ID":"4b2ea88a-8d94-4d82-9988-debfd7a1dc22","Type":"ContainerDied","Data":"988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75"} Jul 14 18:37:44.605619 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:44.605602 2579 scope.go:117] "RemoveContainer" containerID="988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75" Jul 14 18:37:44.606544 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:44.606518 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:37:44.606648 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:44.606606 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:37:44.606648 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:44.606640 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:37:45.617939 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:45.617904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" event={"ID":"4b2ea88a-8d94-4d82-9988-debfd7a1dc22","Type":"ContainerStarted","Data":"aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f"} Jul 14 18:37:46.367492 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:46.367458 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:46.367773 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:46.367749 2579 patch_prober.go:28] interesting pod/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 14 18:37:46.367866 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:46.367803 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 14 18:37:47.367485 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:47.367446 2579 patch_prober.go:28] interesting pod/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 14 18:37:47.367874 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:47.367516 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 14 18:37:48.367101 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:48.367067 2579 patch_prober.go:28] interesting pod/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 14 18:37:48.367292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:48.367120 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 14 18:37:49.370431 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:49.370401 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:49.370849 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:49.370746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:37:49.371448 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:37:49.371429 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:38:57.050991 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:38:57.050915 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv"] Jul 14 18:38:57.051463 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:38:57.051140 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" containerID="cri-o://aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f" gracePeriod=30 Jul 14 18:39:02.332982 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.332950 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:39:02.438371 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438327 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-podinfo\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.438616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438426 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-data\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.438616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438467 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istiod-ca-cert\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.438616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438516 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-certs\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.438616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438548 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-token\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.438616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-credential-socket\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.438888 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438649 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-envoy\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.438888 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438680 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-socket\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.438888 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438715 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9lsc\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-kube-api-access-z9lsc\") pod \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\" (UID: \"4b2ea88a-8d94-4d82-9988-debfd7a1dc22\") " Jul 14 18:39:02.439037 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438871 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:39:02.439037 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.438941 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:39:02.439143 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.439048 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:02.439143 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.439069 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:02.439249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.439150 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:39:02.439249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.439232 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:39:02.439473 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.439441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:39:02.441186 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.441157 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:39:02.441301 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.441195 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:39:02.441301 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.441212 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-kube-api-access-z9lsc" (OuterVolumeSpecName: "kube-api-access-z9lsc") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "kube-api-access-z9lsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:39:02.441842 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.441804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b2ea88a-8d94-4d82-9988-debfd7a1dc22" (UID: "4b2ea88a-8d94-4d82-9988-debfd7a1dc22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:39:02.539628 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.539589 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:02.539628 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.539627 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:02.539818 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.539643 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:02.539818 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.539656 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:02.539818 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.539667 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:02.539818 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.539679 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:02.539818 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:02.539692 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9lsc\" (UniqueName: \"kubernetes.io/projected/4b2ea88a-8d94-4d82-9988-debfd7a1dc22-kube-api-access-z9lsc\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:39:03.032195 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.032146 2579 generic.go:356] "Generic (PLEG): container finished" podID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerID="aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f" exitCode=0 Jul 14 18:39:03.032366 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.032254 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" Jul 14 18:39:03.032366 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.032284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" event={"ID":"4b2ea88a-8d94-4d82-9988-debfd7a1dc22","Type":"ContainerDied","Data":"aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f"} Jul 14 18:39:03.032452 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.032384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv" event={"ID":"4b2ea88a-8d94-4d82-9988-debfd7a1dc22","Type":"ContainerDied","Data":"565475a94e42be377fbb3541b6d7a07a8464ef3eca8111e7502ca75d98481410"} Jul 14 18:39:03.032452 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.032406 2579 scope.go:117] "RemoveContainer" containerID="aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f" Jul 14 18:39:03.046045 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.046022 2579 scope.go:117] "RemoveContainer" containerID="988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75" Jul 14 18:39:03.056791 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.056747 2579 scope.go:117] "RemoveContainer" containerID="aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f" Jul 14 18:39:03.057073 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:39:03.057045 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f\": container with ID starting with aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f not found: ID does not exist" containerID="aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f" Jul 14 18:39:03.057153 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.057087 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f"} err="failed to get container status \"aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f\": rpc error: code = NotFound desc = could not find container \"aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f\": container with ID starting with aba8fccdf8caed52effcc043ac60e5486aaec78f74ae2a4d1b4cdffb81d4c75f not found: ID does not exist" Jul 14 18:39:03.057153 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.057116 2579 scope.go:117] "RemoveContainer" containerID="988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75" Jul 14 18:39:03.057488 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:39:03.057462 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75\": container with ID starting with 988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75 not found: ID does not exist" containerID="988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75" Jul 14 18:39:03.057590 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.057495 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75"} err="failed to get container status \"988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75\": rpc error: code = NotFound desc = could not find container \"988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75\": container with ID starting with 988a83bb408e4ecc3ffaceab177f352b43b4a1048eac8acb51326802d1db0c75 not found: ID does not exist" Jul 14 18:39:03.058123 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.058101 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv"] Jul 14 18:39:03.062804 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.062778 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e2e80b9d-openshift-default-59fdc899f-5b8xv"] Jul 14 18:39:03.208478 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:39:03.208436 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" path="/var/lib/kubelet/pods/4b2ea88a-8d94-4d82-9988-debfd7a1dc22/volumes" Jul 14 18:40:01.181187 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:40:01.181160 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:40:01.187031 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:40:01.187009 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:41:57.352535 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.352455 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l"] Jul 14 18:41:57.352992 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.352810 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" Jul 14 18:41:57.352992 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.352823 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" Jul 14 18:41:57.352992 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.352832 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" Jul 14 18:41:57.352992 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.352838 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" Jul 14 18:41:57.352992 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.352897 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" Jul 14 18:41:57.352992 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.352903 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2ea88a-8d94-4d82-9988-debfd7a1dc22" containerName="istio-proxy" Jul 14 18:41:57.355793 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.355776 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.358625 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.358601 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c99ca1c1\"/\"kube-root-ca.crt\"" Jul 14 18:41:57.359609 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.359592 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c99ca1c1\"/\"openshift-service-ca.crt\"" Jul 14 18:41:57.359780 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.359761 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c99ca1c1\"/\"e2e-lifecycle-model-c99ca1c1-kserve-self-signed-certs\"" Jul 14 18:41:57.359981 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.359961 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c99ca1c1\"/\"default-dockercfg-72g9g\"" Jul 14 18:41:57.365772 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.365752 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l"] Jul 14 18:41:57.427237 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.427203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-home\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.427389 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.427259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-model-cache\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.427389 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.427291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bafe12ce-0f1f-4007-9759-a3241e58bde2-tls-certs\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.427389 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.427375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-dshm\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.427498 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.427403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-tmp-dir\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.427498 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.427422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhs6n\" (UniqueName: \"kubernetes.io/projected/bafe12ce-0f1f-4007-9759-a3241e58bde2-kube-api-access-zhs6n\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528126 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-model-cache\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528126 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bafe12ce-0f1f-4007-9759-a3241e58bde2-tls-certs\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528393 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-dshm\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528393 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-tmp-dir\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528393 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs6n\" (UniqueName: \"kubernetes.io/projected/bafe12ce-0f1f-4007-9759-a3241e58bde2-kube-api-access-zhs6n\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528393 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-home\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528644 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-model-cache\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528688 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-tmp-dir\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.528733 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.528717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-home\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.530538 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.530518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-dshm\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.530701 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.530685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bafe12ce-0f1f-4007-9759-a3241e58bde2-tls-certs\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.538880 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.538858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhs6n\" (UniqueName: \"kubernetes.io/projected/bafe12ce-0f1f-4007-9759-a3241e58bde2-kube-api-access-zhs6n\") pod \"e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.666616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.666505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:41:57.803397 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.803281 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l"] Jul 14 18:41:57.805488 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:41:57.805464 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbafe12ce_0f1f_4007_9759_a3241e58bde2.slice/crio-69f72d8195ffb05269397bb06e625bcd163abc29a47133348a807ee7dea375bc WatchSource:0}: Error finding container 69f72d8195ffb05269397bb06e625bcd163abc29a47133348a807ee7dea375bc: Status 404 returned error can't find the container with id 69f72d8195ffb05269397bb06e625bcd163abc29a47133348a807ee7dea375bc Jul 14 18:41:57.921745 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.921662 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" event={"ID":"bafe12ce-0f1f-4007-9759-a3241e58bde2","Type":"ContainerStarted","Data":"b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e"} Jul 14 18:41:57.921745 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.921704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" event={"ID":"bafe12ce-0f1f-4007-9759-a3241e58bde2","Type":"ContainerStarted","Data":"69f72d8195ffb05269397bb06e625bcd163abc29a47133348a807ee7dea375bc"} Jul 14 18:41:57.941078 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:41:57.940984 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" podStartSLOduration=0.940965245 podStartE2EDuration="940.965245ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:41:57.940886232 +0000 UTC m=+1917.372741666" watchObservedRunningTime="2026-07-14 18:41:57.940965245 +0000 UTC m=+1917.372820683" Jul 14 18:42:07.666916 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:42:07.666873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:42:07.667298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:42:07.666938 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:42:07.679492 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:42:07.679463 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:42:07.984157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:42:07.984133 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:42:47.582037 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:42:47.582003 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l"] Jul 14 18:43:02.602502 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.602398 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" podUID="bafe12ce-0f1f-4007-9759-a3241e58bde2" containerName="main" containerID="cri-o://b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e" gracePeriod=45 Jul 14 18:43:02.849448 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.849420 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:43:02.877694 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.877616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-home\") pod \"bafe12ce-0f1f-4007-9759-a3241e58bde2\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " Jul 14 18:43:02.877694 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.877658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-model-cache\") pod \"bafe12ce-0f1f-4007-9759-a3241e58bde2\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " Jul 14 18:43:02.877926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.877697 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-dshm\") pod \"bafe12ce-0f1f-4007-9759-a3241e58bde2\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " Jul 14 18:43:02.877926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.877735 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhs6n\" (UniqueName: \"kubernetes.io/projected/bafe12ce-0f1f-4007-9759-a3241e58bde2-kube-api-access-zhs6n\") pod \"bafe12ce-0f1f-4007-9759-a3241e58bde2\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " Jul 14 18:43:02.877926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.877829 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bafe12ce-0f1f-4007-9759-a3241e58bde2-tls-certs\") pod \"bafe12ce-0f1f-4007-9759-a3241e58bde2\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " Jul 14 18:43:02.877926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.877872 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-tmp-dir\") pod \"bafe12ce-0f1f-4007-9759-a3241e58bde2\" (UID: \"bafe12ce-0f1f-4007-9759-a3241e58bde2\") " Jul 14 18:43:02.877926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.877883 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-home" (OuterVolumeSpecName: "home") pod "bafe12ce-0f1f-4007-9759-a3241e58bde2" (UID: "bafe12ce-0f1f-4007-9759-a3241e58bde2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:02.878193 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.877998 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-model-cache" (OuterVolumeSpecName: "model-cache") pod "bafe12ce-0f1f-4007-9759-a3241e58bde2" (UID: "bafe12ce-0f1f-4007-9759-a3241e58bde2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:02.878193 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.878149 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:43:02.878193 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.878167 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:43:02.878665 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.878614 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bafe12ce-0f1f-4007-9759-a3241e58bde2" (UID: "bafe12ce-0f1f-4007-9759-a3241e58bde2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:02.880491 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.880462 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafe12ce-0f1f-4007-9759-a3241e58bde2-kube-api-access-zhs6n" (OuterVolumeSpecName: "kube-api-access-zhs6n") pod "bafe12ce-0f1f-4007-9759-a3241e58bde2" (UID: "bafe12ce-0f1f-4007-9759-a3241e58bde2"). InnerVolumeSpecName "kube-api-access-zhs6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:43:02.880491 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.880486 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-dshm" (OuterVolumeSpecName: "dshm") pod "bafe12ce-0f1f-4007-9759-a3241e58bde2" (UID: "bafe12ce-0f1f-4007-9759-a3241e58bde2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:02.880715 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.880699 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bafe12ce-0f1f-4007-9759-a3241e58bde2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bafe12ce-0f1f-4007-9759-a3241e58bde2" (UID: "bafe12ce-0f1f-4007-9759-a3241e58bde2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:43:02.979293 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.979259 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:43:02.979293 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.979287 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhs6n\" (UniqueName: \"kubernetes.io/projected/bafe12ce-0f1f-4007-9759-a3241e58bde2-kube-api-access-zhs6n\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:43:02.979293 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.979298 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bafe12ce-0f1f-4007-9759-a3241e58bde2-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:43:02.979526 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:02.979307 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bafe12ce-0f1f-4007-9759-a3241e58bde2-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:43:03.255056 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.255020 2579 generic.go:356] "Generic (PLEG): container finished" podID="bafe12ce-0f1f-4007-9759-a3241e58bde2" containerID="b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e" exitCode=0 Jul 14 18:43:03.255214 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.255063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" event={"ID":"bafe12ce-0f1f-4007-9759-a3241e58bde2","Type":"ContainerDied","Data":"b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e"} Jul 14 18:43:03.255214 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.255108 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" event={"ID":"bafe12ce-0f1f-4007-9759-a3241e58bde2","Type":"ContainerDied","Data":"69f72d8195ffb05269397bb06e625bcd163abc29a47133348a807ee7dea375bc"} Jul 14 18:43:03.255214 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.255073 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l" Jul 14 18:43:03.255214 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.255129 2579 scope.go:117] "RemoveContainer" containerID="b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e" Jul 14 18:43:03.266241 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.266194 2579 scope.go:117] "RemoveContainer" containerID="b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e" Jul 14 18:43:03.266606 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:43:03.266579 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e\": container with ID starting with b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e not found: ID does not exist" containerID="b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e" Jul 14 18:43:03.266732 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.266613 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e"} err="failed to get container status \"b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e\": rpc error: code = NotFound desc = could not find container \"b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e\": container with ID starting with b90f3f6d432441c2ca66e0526b5414c9a4c9994e703047eabe13268a3196629e not found: ID does not exist" Jul 14 18:43:03.275622 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.275599 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l"] Jul 14 18:43:03.280681 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:03.280659 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c99ca1c1/e2e-lifecycle-model-c99ca1c1-kserve-5ddf8b455f-qlj6l"] Jul 14 18:43:05.207201 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:43:05.207166 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bafe12ce-0f1f-4007-9759-a3241e58bde2" path="/var/lib/kubelet/pods/bafe12ce-0f1f-4007-9759-a3241e58bde2/volumes" Jul 14 18:44:44.661205 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.661120 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597"] Jul 14 18:44:44.661668 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.661469 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bafe12ce-0f1f-4007-9759-a3241e58bde2" containerName="main" Jul 14 18:44:44.661668 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.661480 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bafe12ce-0f1f-4007-9759-a3241e58bde2" containerName="main" Jul 14 18:44:44.661668 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.661571 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bafe12ce-0f1f-4007-9759-a3241e58bde2" containerName="main" Jul 14 18:44:44.664489 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.664467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667153 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667130 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7965f3-openshift-default-dockercfg-chvhk\"" Jul 14 18:44:44.667261 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-envoy\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667261 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351fd327-ea78-4e75-879e-79f6992bb721-istio-podinfo\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667261 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-credential-socket\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667425 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-socket\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667425 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351fd327-ea78-4e75-879e-79f6992bb721-istiod-ca-cert\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667425 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-istio-token\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667425 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-data\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667682 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-certs\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.667682 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.667473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrs6h\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-kube-api-access-vrs6h\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.677294 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.677270 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597"] Jul 14 18:44:44.768010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.767978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-data\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.768010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.768012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-certs\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.768262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.768037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrs6h\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-kube-api-access-vrs6h\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.768262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.768059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-envoy\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.768262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.768080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351fd327-ea78-4e75-879e-79f6992bb721-istio-podinfo\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.768262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.768097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-credential-socket\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.768262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.768125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-socket\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.768262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.768143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351fd327-ea78-4e75-879e-79f6992bb721-istiod-ca-cert\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.768262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.768159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-istio-token\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.769262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.769118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-certs\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.769387 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.769305 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-credential-socket\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.769387 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.769332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-socket\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.770185 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.770164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351fd327-ea78-4e75-879e-79f6992bb721-istiod-ca-cert\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.770657 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.770106 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-data\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.771973 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.771948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-envoy\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.772415 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.772388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351fd327-ea78-4e75-879e-79f6992bb721-istio-podinfo\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.783048 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.782996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-istio-token\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.783692 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.783667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrs6h\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-kube-api-access-vrs6h\") pod \"e2e-inf-b-7965f3-openshift-default-b565fb887-9p597\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:44.975011 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:44.974981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:45.112381 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:45.112357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597"] Jul 14 18:44:45.114120 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:44:45.114094 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351fd327_ea78_4e75_879e_79f6992bb721.slice/crio-82f5b680e17aea310c48ab96650984866b1bb1d221cdb9b669c43258833f8b25 WatchSource:0}: Error finding container 82f5b680e17aea310c48ab96650984866b1bb1d221cdb9b669c43258833f8b25: Status 404 returned error can't find the container with id 82f5b680e17aea310c48ab96650984866b1bb1d221cdb9b669c43258833f8b25 Jul 14 18:44:45.116245 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:45.116212 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:44:45.116328 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:45.116275 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:44:45.116328 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:45.116302 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:44:45.790607 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:45.790577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" event={"ID":"351fd327-ea78-4e75-879e-79f6992bb721","Type":"ContainerStarted","Data":"f02e07983114b5f222d4a731e2e4fdc6ad98cdf2a7cfb11ac63fda59b1f7fce0"} Jul 14 18:44:45.790976 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:45.790628 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" event={"ID":"351fd327-ea78-4e75-879e-79f6992bb721","Type":"ContainerStarted","Data":"82f5b680e17aea310c48ab96650984866b1bb1d221cdb9b669c43258833f8b25"} Jul 14 18:44:45.822425 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:45.822379 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podStartSLOduration=1.8223638 podStartE2EDuration="1.8223638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:44:45.820649885 +0000 UTC m=+2085.252505320" watchObservedRunningTime="2026-07-14 18:44:45.8223638 +0000 UTC m=+2085.254219234" Jul 14 18:44:45.975896 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:45.975860 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:46.980147 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:46.980115 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:47.800470 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:47.800439 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:44:47.801607 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:44:47.801588 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:45:01.217322 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:01.217290 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:45:01.222798 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:01.222779 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:45:03.770610 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:03.769531 2579 patch_prober.go:28] interesting pod/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:33532->10.133.0.52:15021: read: connection reset by peer" start-of-body= Jul 14 18:45:03.770610 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:03.769622 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:33532->10.133.0.52:15021: read: connection reset by peer" Jul 14 18:45:03.897848 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:03.897816 2579 generic.go:356] "Generic (PLEG): container finished" podID="351fd327-ea78-4e75-879e-79f6992bb721" containerID="f02e07983114b5f222d4a731e2e4fdc6ad98cdf2a7cfb11ac63fda59b1f7fce0" exitCode=0 Jul 14 18:45:03.898007 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:03.897885 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" event={"ID":"351fd327-ea78-4e75-879e-79f6992bb721","Type":"ContainerDied","Data":"f02e07983114b5f222d4a731e2e4fdc6ad98cdf2a7cfb11ac63fda59b1f7fce0"} Jul 14 18:45:03.898290 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:03.898266 2579 scope.go:117] "RemoveContainer" containerID="f02e07983114b5f222d4a731e2e4fdc6ad98cdf2a7cfb11ac63fda59b1f7fce0" Jul 14 18:45:03.899250 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:03.899224 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:45:03.899309 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:03.899288 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:45:03.899354 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:03.899318 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:45:04.905603 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:04.905570 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" event={"ID":"351fd327-ea78-4e75-879e-79f6992bb721","Type":"ContainerStarted","Data":"38530f019d5470844322e038f1e4810ce89f2f39b7061bc1e0f76f7ac6828ba6"} Jul 14 18:45:04.975631 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:04.975594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:45:04.975894 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:04.975874 2579 patch_prober.go:28] interesting pod/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 14 18:45:04.975940 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:04.975921 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 14 18:45:05.975803 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:05.975771 2579 patch_prober.go:28] interesting pod/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 14 18:45:05.976171 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:05.975839 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 14 18:45:06.975786 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:06.975746 2579 patch_prober.go:28] interesting pod/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 14 18:45:06.975970 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:06.975804 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 14 18:45:07.975339 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:07.975308 2579 patch_prober.go:28] interesting pod/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 14 18:45:07.975509 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:07.975360 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 14 18:45:08.976381 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:08.976344 2579 patch_prober.go:28] interesting pod/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 14 18:45:08.976803 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:08.976399 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 14 18:45:09.978855 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:09.978823 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:45:09.979305 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:09.979065 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:45:09.979952 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:09.979932 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:45:38.062608 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.062565 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j"] Jul 14 18:45:38.066482 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.066461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.069371 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.069345 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7965f3\"/\"kube-root-ca.crt\"" Jul 14 18:45:38.070674 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.070651 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7965f3\"/\"test-model-7965f3-kserve-self-signed-certs\"" Jul 14 18:45:38.070798 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.070651 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7965f3\"/\"default-dockercfg-xs2nt\"" Jul 14 18:45:38.070798 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.070717 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7965f3\"/\"openshift-service-ca.crt\"" Jul 14 18:45:38.075056 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.075031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j"] Jul 14 18:45:38.131693 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.131660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-model-cache\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.131876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.131714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-dshm\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.131876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.131763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hnb\" (UniqueName: \"kubernetes.io/projected/b6342142-eebc-4c5b-b827-dcd4d7979f9a-kube-api-access-64hnb\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.131876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.131811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tls-certs\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.131876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.131852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tmp-dir\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.132037 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.131931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-home\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.232357 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.232318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-home\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.232357 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.232359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-model-cache\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.232622 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.232544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-dshm\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.232622 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.232604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hnb\" (UniqueName: \"kubernetes.io/projected/b6342142-eebc-4c5b-b827-dcd4d7979f9a-kube-api-access-64hnb\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.232753 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.232732 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-home\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.232796 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.232780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tls-certs\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.232832 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.232802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-model-cache\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.232876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.232834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tmp-dir\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.233128 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.233097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tmp-dir\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.234844 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.234820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-dshm\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.235062 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.235045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tls-certs\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.245685 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.245663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hnb\" (UniqueName: \"kubernetes.io/projected/b6342142-eebc-4c5b-b827-dcd4d7979f9a-kube-api-access-64hnb\") pod \"test-model-7965f3-kserve-7477ff886f-2sn8j\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.377233 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.377148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:38.513422 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:38.513390 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j"] Jul 14 18:45:38.514302 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:45:38.514269 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6342142_eebc_4c5b_b827_dcd4d7979f9a.slice/crio-01286809e930e863a13b6d3cdbde489b6bc5c18792d4da6709f5ee1a4b04fc24 WatchSource:0}: Error finding container 01286809e930e863a13b6d3cdbde489b6bc5c18792d4da6709f5ee1a4b04fc24: Status 404 returned error can't find the container with id 01286809e930e863a13b6d3cdbde489b6bc5c18792d4da6709f5ee1a4b04fc24 Jul 14 18:45:39.086161 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:39.086126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" event={"ID":"b6342142-eebc-4c5b-b827-dcd4d7979f9a","Type":"ContainerStarted","Data":"15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d"} Jul 14 18:45:39.086582 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:39.086170 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" event={"ID":"b6342142-eebc-4c5b-b827-dcd4d7979f9a","Type":"ContainerStarted","Data":"01286809e930e863a13b6d3cdbde489b6bc5c18792d4da6709f5ee1a4b04fc24"} Jul 14 18:45:39.112017 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:39.111963 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" podStartSLOduration=1.11194632 podStartE2EDuration="1.11194632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:45:39.110766104 +0000 UTC m=+2138.542621559" watchObservedRunningTime="2026-07-14 18:45:39.11194632 +0000 UTC m=+2138.543801764" Jul 14 18:45:48.378259 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:48.378228 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:48.378790 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:48.378481 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:48.390926 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:48.390900 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:45:49.147064 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:45:49.147034 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:46:13.029906 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:13.029872 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j"] Jul 14 18:46:28.053044 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.052981 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" podUID="b6342142-eebc-4c5b-b827-dcd4d7979f9a" containerName="main" containerID="cri-o://15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d" gracePeriod=45 Jul 14 18:46:28.303970 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.303909 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:46:28.341446 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.341410 2579 generic.go:356] "Generic (PLEG): container finished" podID="b6342142-eebc-4c5b-b827-dcd4d7979f9a" containerID="15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d" exitCode=0 Jul 14 18:46:28.341446 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.341447 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" Jul 14 18:46:28.341723 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.341458 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" event={"ID":"b6342142-eebc-4c5b-b827-dcd4d7979f9a","Type":"ContainerDied","Data":"15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d"} Jul 14 18:46:28.341723 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.341500 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j" event={"ID":"b6342142-eebc-4c5b-b827-dcd4d7979f9a","Type":"ContainerDied","Data":"01286809e930e863a13b6d3cdbde489b6bc5c18792d4da6709f5ee1a4b04fc24"} Jul 14 18:46:28.341723 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.341518 2579 scope.go:117] "RemoveContainer" containerID="15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d" Jul 14 18:46:28.350442 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.350420 2579 scope.go:117] "RemoveContainer" containerID="15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d" Jul 14 18:46:28.350828 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:46:28.350805 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d\": container with ID starting with 15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d not found: ID does not exist" containerID="15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d" Jul 14 18:46:28.350923 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.350837 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d"} err="failed to get container status \"15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d\": rpc error: code = NotFound desc = could not find container \"15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d\": container with ID starting with 15e22675aa12d38e3307454e65b0f93d9985c1a7f0ab42926b2c6c7caef6cf4d not found: ID does not exist" Jul 14 18:46:28.378034 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378010 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64hnb\" (UniqueName: \"kubernetes.io/projected/b6342142-eebc-4c5b-b827-dcd4d7979f9a-kube-api-access-64hnb\") pod \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " Jul 14 18:46:28.378191 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378044 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-dshm\") pod \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " Jul 14 18:46:28.378191 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378107 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tls-certs\") pod \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " Jul 14 18:46:28.378191 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378130 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-home\") pod \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " Jul 14 18:46:28.378191 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378168 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-model-cache\") pod \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " Jul 14 18:46:28.378390 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tmp-dir\") pod \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\" (UID: \"b6342142-eebc-4c5b-b827-dcd4d7979f9a\") " Jul 14 18:46:28.378494 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378460 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-home" (OuterVolumeSpecName: "home") pod "b6342142-eebc-4c5b-b827-dcd4d7979f9a" (UID: "b6342142-eebc-4c5b-b827-dcd4d7979f9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:28.378646 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378490 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6342142-eebc-4c5b-b827-dcd4d7979f9a" (UID: "b6342142-eebc-4c5b-b827-dcd4d7979f9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:28.378646 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.378602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6342142-eebc-4c5b-b827-dcd4d7979f9a" (UID: "b6342142-eebc-4c5b-b827-dcd4d7979f9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:28.380235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.380212 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6342142-eebc-4c5b-b827-dcd4d7979f9a-kube-api-access-64hnb" (OuterVolumeSpecName: "kube-api-access-64hnb") pod "b6342142-eebc-4c5b-b827-dcd4d7979f9a" (UID: "b6342142-eebc-4c5b-b827-dcd4d7979f9a"). InnerVolumeSpecName "kube-api-access-64hnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:46:28.380235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.380217 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-dshm" (OuterVolumeSpecName: "dshm") pod "b6342142-eebc-4c5b-b827-dcd4d7979f9a" (UID: "b6342142-eebc-4c5b-b827-dcd4d7979f9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:28.380235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.380231 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6342142-eebc-4c5b-b827-dcd4d7979f9a" (UID: "b6342142-eebc-4c5b-b827-dcd4d7979f9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:46:28.479212 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.479180 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:46:28.479212 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.479204 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:46:28.479212 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.479213 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:46:28.479212 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.479221 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:46:28.479512 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.479230 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hnb\" (UniqueName: \"kubernetes.io/projected/b6342142-eebc-4c5b-b827-dcd4d7979f9a-kube-api-access-64hnb\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:46:28.479512 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.479240 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6342142-eebc-4c5b-b827-dcd4d7979f9a-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:46:28.667286 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.667255 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j"] Jul 14 18:46:28.672355 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:28.672331 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7965f3/test-model-7965f3-kserve-7477ff886f-2sn8j"] Jul 14 18:46:29.207308 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:46:29.207270 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6342142-eebc-4c5b-b827-dcd4d7979f9a" path="/var/lib/kubelet/pods/b6342142-eebc-4c5b-b827-dcd4d7979f9a/volumes" Jul 14 18:47:05.364398 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:05.364358 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597"] Jul 14 18:47:05.364876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:05.364604 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" containerID="cri-o://38530f019d5470844322e038f1e4810ce89f2f39b7061bc1e0f76f7ac6828ba6" gracePeriod=30 Jul 14 18:47:09.982133 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:09.982086 2579 patch_prober.go:28] interesting pod/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:47:09.982621 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:09.982149 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:47:10.572423 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.572391 2579 generic.go:356] "Generic (PLEG): container finished" podID="351fd327-ea78-4e75-879e-79f6992bb721" containerID="38530f019d5470844322e038f1e4810ce89f2f39b7061bc1e0f76f7ac6828ba6" exitCode=0 Jul 14 18:47:10.572423 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.572418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" event={"ID":"351fd327-ea78-4e75-879e-79f6992bb721","Type":"ContainerDied","Data":"38530f019d5470844322e038f1e4810ce89f2f39b7061bc1e0f76f7ac6828ba6"} Jul 14 18:47:10.572719 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.572486 2579 scope.go:117] "RemoveContainer" containerID="f02e07983114b5f222d4a731e2e4fdc6ad98cdf2a7cfb11ac63fda59b1f7fce0" Jul 14 18:47:10.625142 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.625084 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:47:10.726091 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726054 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrs6h\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-kube-api-access-vrs6h\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726239 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726106 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-data\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726239 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726125 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-istio-token\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726239 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726159 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-credential-socket\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726239 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726196 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-envoy\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726239 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-certs\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726508 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-socket\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726508 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726308 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351fd327-ea78-4e75-879e-79f6992bb721-istio-podinfo\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726508 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726337 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351fd327-ea78-4e75-879e-79f6992bb721-istiod-ca-cert\") pod \"351fd327-ea78-4e75-879e-79f6992bb721\" (UID: \"351fd327-ea78-4e75-879e-79f6992bb721\") " Jul 14 18:47:10.726649 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726534 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:47:10.726649 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726547 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:47:10.726902 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726856 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:47:10.727010 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.726917 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351fd327-ea78-4e75-879e-79f6992bb721-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:47:10.727086 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.727062 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-data" (OuterVolumeSpecName: "istio-data") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:47:10.728663 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.728634 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-istio-token" (OuterVolumeSpecName: "istio-token") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:47:10.728760 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.728690 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/351fd327-ea78-4e75-879e-79f6992bb721-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:47:10.728836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.728814 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:47:10.728874 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.728822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-kube-api-access-vrs6h" (OuterVolumeSpecName: "kube-api-access-vrs6h") pod "351fd327-ea78-4e75-879e-79f6992bb721" (UID: "351fd327-ea78-4e75-879e-79f6992bb721"). InnerVolumeSpecName "kube-api-access-vrs6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:47:10.828068 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828035 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:10.828068 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828063 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:10.828068 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828073 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:10.828298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828082 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351fd327-ea78-4e75-879e-79f6992bb721-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:10.828298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828091 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351fd327-ea78-4e75-879e-79f6992bb721-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:10.828298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828099 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrs6h\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-kube-api-access-vrs6h\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:10.828298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828108 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:10.828298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828116 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351fd327-ea78-4e75-879e-79f6992bb721-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:10.828298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:10.828123 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351fd327-ea78-4e75-879e-79f6992bb721-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:47:11.578806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:11.578727 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" Jul 14 18:47:11.578806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:11.578763 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597" event={"ID":"351fd327-ea78-4e75-879e-79f6992bb721","Type":"ContainerDied","Data":"82f5b680e17aea310c48ab96650984866b1bb1d221cdb9b669c43258833f8b25"} Jul 14 18:47:11.579245 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:11.578817 2579 scope.go:117] "RemoveContainer" containerID="38530f019d5470844322e038f1e4810ce89f2f39b7061bc1e0f76f7ac6828ba6" Jul 14 18:47:11.600253 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:11.600224 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597"] Jul 14 18:47:11.604480 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:11.604457 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7965f3-openshift-default-b565fb887-9p597"] Jul 14 18:47:13.207713 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:13.207679 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351fd327-ea78-4e75-879e-79f6992bb721" path="/var/lib/kubelet/pods/351fd327-ea78-4e75-879e-79f6992bb721/volumes" Jul 14 18:47:14.511001 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.510967 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p"] Jul 14 18:47:14.511457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.511317 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" Jul 14 18:47:14.511457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.511333 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" Jul 14 18:47:14.511457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.511346 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6342142-eebc-4c5b-b827-dcd4d7979f9a" containerName="main" Jul 14 18:47:14.511457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.511352 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6342142-eebc-4c5b-b827-dcd4d7979f9a" containerName="main" Jul 14 18:47:14.511457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.511417 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" Jul 14 18:47:14.511457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.511427 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6342142-eebc-4c5b-b827-dcd4d7979f9a" containerName="main" Jul 14 18:47:14.511457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.511436 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" Jul 14 18:47:14.515948 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.515916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.518670 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.518645 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fde6bc-openshift-default-dockercfg-d4kgs\"" Jul 14 18:47:14.527456 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.527426 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p"] Jul 14 18:47:14.661982 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.661946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-certs\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.661982 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.661985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zh75\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-kube-api-access-7zh75\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.662228 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.662007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-data\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.662228 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.662034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-token\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.662228 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.662094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-credential-socket\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.662228 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.662119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-envoy\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.662228 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.662138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b286750-826d-41b0-9c87-709dfaa6fb7a-istiod-ca-cert\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.662228 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.662156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-podinfo\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.662430 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.662264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-socket\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763394 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-socket\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763394 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-certs\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763394 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zh75\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-kube-api-access-7zh75\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763394 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-data\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763776 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-token\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763776 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-credential-socket\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763776 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-envoy\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763776 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b286750-826d-41b0-9c87-709dfaa6fb7a-istiod-ca-cert\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.763776 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-podinfo\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.764095 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-data\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.764095 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-socket\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.764095 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.763869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-certs\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.764235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.764139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-credential-socket\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.764431 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.764415 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b286750-826d-41b0-9c87-709dfaa6fb7a-istiod-ca-cert\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.766563 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.766531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-envoy\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.766863 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.766840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-podinfo\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.772825 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.772794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-token\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.773080 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.773057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zh75\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-kube-api-access-7zh75\") pod \"e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.826692 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.826653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:14.975685 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.975654 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p"] Jul 14 18:47:14.977128 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:47:14.977100 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b286750_826d_41b0_9c87_709dfaa6fb7a.slice/crio-518ffc07149e23cf6bc14df02859b7bbd75083ba96086b8ebce31b5cf32f7fb6 WatchSource:0}: Error finding container 518ffc07149e23cf6bc14df02859b7bbd75083ba96086b8ebce31b5cf32f7fb6: Status 404 returned error can't find the container with id 518ffc07149e23cf6bc14df02859b7bbd75083ba96086b8ebce31b5cf32f7fb6 Jul 14 18:47:14.979262 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.979227 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:47:14.979371 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.979306 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:47:14.979371 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:14.979350 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:47:15.603746 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:15.603704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" event={"ID":"5b286750-826d-41b0-9c87-709dfaa6fb7a","Type":"ContainerStarted","Data":"30a5c334749cf2e2c98839d98f7dc7d2d6b4f74c07d76434c5fe6ca9e6c086a7"} Jul 14 18:47:15.604206 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:15.603765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" event={"ID":"5b286750-826d-41b0-9c87-709dfaa6fb7a","Type":"ContainerStarted","Data":"518ffc07149e23cf6bc14df02859b7bbd75083ba96086b8ebce31b5cf32f7fb6"} Jul 14 18:47:15.630068 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:15.630019 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" podStartSLOduration=1.630004398 podStartE2EDuration="1.630004398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:47:15.627905287 +0000 UTC m=+2235.059760722" watchObservedRunningTime="2026-07-14 18:47:15.630004398 +0000 UTC m=+2235.061859831" Jul 14 18:47:15.827279 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:15.827237 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:16.832281 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:16.832251 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:17.614246 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:17.614211 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:17.615110 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:17.615091 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:36.712905 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:36.712872 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerID="30a5c334749cf2e2c98839d98f7dc7d2d6b4f74c07d76434c5fe6ca9e6c086a7" exitCode=0 Jul 14 18:47:36.713336 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:36.712944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" event={"ID":"5b286750-826d-41b0-9c87-709dfaa6fb7a","Type":"ContainerDied","Data":"30a5c334749cf2e2c98839d98f7dc7d2d6b4f74c07d76434c5fe6ca9e6c086a7"} Jul 14 18:47:36.713336 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:36.713303 2579 scope.go:117] "RemoveContainer" containerID="30a5c334749cf2e2c98839d98f7dc7d2d6b4f74c07d76434c5fe6ca9e6c086a7" Jul 14 18:47:36.714304 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:36.714281 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:47:36.714357 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:36.714345 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:47:36.714400 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:36.714373 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:47:37.719878 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:37.719843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" event={"ID":"5b286750-826d-41b0-9c87-709dfaa6fb7a","Type":"ContainerStarted","Data":"ccddb806ae18a49cea0e44ccad9f1f8b949dc1a712f210e839f1f4be8e01ca3a"} Jul 14 18:47:37.826974 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:37.826942 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:37.827156 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:37.827134 2579 patch_prober.go:28] interesting pod/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:37.827201 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:37.827182 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 14 18:47:38.827991 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:38.827955 2579 patch_prober.go:28] interesting pod/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:38.828477 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:38.828018 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 14 18:47:39.827693 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:39.827641 2579 patch_prober.go:28] interesting pod/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:39.827868 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:39.827713 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 14 18:47:40.827966 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:40.827927 2579 patch_prober.go:28] interesting pod/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:40.828371 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:40.828003 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 14 18:47:41.830991 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:41.830961 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:41.831486 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:41.831452 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:47:41.832340 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:47:41.832320 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:48:22.771582 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:22.771536 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p"] Jul 14 18:48:22.772058 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:22.771782 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" containerID="cri-o://ccddb806ae18a49cea0e44ccad9f1f8b949dc1a712f210e839f1f4be8e01ca3a" gracePeriod=30 Jul 14 18:48:26.834539 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:26.834498 2579 patch_prober.go:28] interesting pod/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:48:26.835070 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:26.834595 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:48:27.999744 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:27.999706 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerID="ccddb806ae18a49cea0e44ccad9f1f8b949dc1a712f210e839f1f4be8e01ca3a" exitCode=0 Jul 14 18:48:28.000085 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:27.999771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" event={"ID":"5b286750-826d-41b0-9c87-709dfaa6fb7a","Type":"ContainerDied","Data":"ccddb806ae18a49cea0e44ccad9f1f8b949dc1a712f210e839f1f4be8e01ca3a"} Jul 14 18:48:28.000085 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:27.999864 2579 scope.go:117] "RemoveContainer" containerID="30a5c334749cf2e2c98839d98f7dc7d2d6b4f74c07d76434c5fe6ca9e6c086a7" Jul 14 18:48:28.019970 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.019945 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:48:28.116411 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116325 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-token\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116411 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-credential-socket\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116411 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116388 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b286750-826d-41b0-9c87-709dfaa6fb7a-istiod-ca-cert\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116411 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116409 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zh75\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-kube-api-access-7zh75\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116763 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116445 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-podinfo\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116763 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116474 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-envoy\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116763 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116511 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-data\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116763 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116578 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-certs\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116763 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116603 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-socket\") pod \"5b286750-826d-41b0-9c87-709dfaa6fb7a\" (UID: \"5b286750-826d-41b0-9c87-709dfaa6fb7a\") " Jul 14 18:48:28.116763 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116649 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:28.117084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116861 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.117084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116864 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:28.117084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.116884 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b286750-826d-41b0-9c87-709dfaa6fb7a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:48:28.117084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.117003 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:28.117638 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.117616 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:28.118713 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.118692 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:48:28.118848 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.118829 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-kube-api-access-7zh75" (OuterVolumeSpecName: "kube-api-access-7zh75") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "kube-api-access-7zh75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:48:28.119122 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.119101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:48:28.119185 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.119123 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b286750-826d-41b0-9c87-709dfaa6fb7a" (UID: "5b286750-826d-41b0-9c87-709dfaa6fb7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:28.218216 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.218175 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.218216 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.218206 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b286750-826d-41b0-9c87-709dfaa6fb7a-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.218216 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.218218 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zh75\" (UniqueName: \"kubernetes.io/projected/5b286750-826d-41b0-9c87-709dfaa6fb7a-kube-api-access-7zh75\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.218468 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.218231 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.218468 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.218243 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.218468 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.218253 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.218468 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.218266 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.218468 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:28.218277 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b286750-826d-41b0-9c87-709dfaa6fb7a-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:48:29.006506 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:29.006471 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" Jul 14 18:48:29.006975 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:29.006507 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p" event={"ID":"5b286750-826d-41b0-9c87-709dfaa6fb7a","Type":"ContainerDied","Data":"518ffc07149e23cf6bc14df02859b7bbd75083ba96086b8ebce31b5cf32f7fb6"} Jul 14 18:48:29.006975 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:29.006586 2579 scope.go:117] "RemoveContainer" containerID="ccddb806ae18a49cea0e44ccad9f1f8b949dc1a712f210e839f1f4be8e01ca3a" Jul 14 18:48:29.043244 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:29.043212 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p"] Jul 14 18:48:29.048468 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:29.048440 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fde6bc-openshift-default-68855c7c6f-xhz9p"] Jul 14 18:48:29.207756 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:29.207723 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" path="/var/lib/kubelet/pods/5b286750-826d-41b0-9c87-709dfaa6fb7a/volumes" Jul 14 18:48:31.808235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.808202 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44"] Jul 14 18:48:31.808709 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.808635 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" Jul 14 18:48:31.808709 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.808655 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="351fd327-ea78-4e75-879e-79f6992bb721" containerName="istio-proxy" Jul 14 18:48:31.808709 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.808679 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" Jul 14 18:48:31.808709 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.808685 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" Jul 14 18:48:31.808916 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.808772 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" Jul 14 18:48:31.808916 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.808783 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" Jul 14 18:48:31.813238 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.813211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.815593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.815572 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-41a771-openshift-default-dockercfg-n2llv\"" Jul 14 18:48:31.826404 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.826376 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44"] Jul 14 18:48:31.850879 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.850848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-podinfo\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.851052 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.850899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-socket\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.851052 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.850923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bvz\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-kube-api-access-84bvz\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.851052 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.850943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-envoy\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.851052 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.850959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-data\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.851218 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.851072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-certs\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.851218 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.851108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istiod-ca-cert\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.851218 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.851143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-credential-socket\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.851218 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.851158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-token\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952372 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-envoy\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952372 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-data\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952638 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-certs\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952638 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istiod-ca-cert\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952638 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-credential-socket\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952638 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-token\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952638 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-podinfo\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952638 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-socket\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.952889 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.952641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bvz\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-kube-api-access-84bvz\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.953290 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.953265 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-data\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.953462 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.953435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-socket\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.953593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.953398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-credential-socket\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.953593 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.953323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-certs\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.953680 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.953622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istiod-ca-cert\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.955355 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.955336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-envoy\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.955457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.955439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-podinfo\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.961978 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.961956 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-token\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:31.962343 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:31.962323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bvz\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-kube-api-access-84bvz\") pod \"e2e-shared-b-41a771-openshift-default-6886556bcf-86f44\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:32.127222 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:32.127133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:32.269097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:32.269072 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44"] Jul 14 18:48:32.271535 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:48:32.271509 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426d9fb0_b6fa_4bc7_9774_60ea67aaf152.slice/crio-088fda1664fffc75895f90d3c0f56372cefcdbfab6db2eae829034ba05a07a0e WatchSource:0}: Error finding container 088fda1664fffc75895f90d3c0f56372cefcdbfab6db2eae829034ba05a07a0e: Status 404 returned error can't find the container with id 088fda1664fffc75895f90d3c0f56372cefcdbfab6db2eae829034ba05a07a0e Jul 14 18:48:32.273749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:32.273714 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:48:32.273852 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:32.273779 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:48:32.273852 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:32.273808 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:48:33.031836 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:33.031802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" event={"ID":"426d9fb0-b6fa-4bc7-9774-60ea67aaf152","Type":"ContainerStarted","Data":"a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12"} Jul 14 18:48:33.032244 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:33.031852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" event={"ID":"426d9fb0-b6fa-4bc7-9774-60ea67aaf152","Type":"ContainerStarted","Data":"088fda1664fffc75895f90d3c0f56372cefcdbfab6db2eae829034ba05a07a0e"} Jul 14 18:48:33.056610 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:33.056538 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" podStartSLOduration=2.056520838 podStartE2EDuration="2.056520838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:48:33.053368335 +0000 UTC m=+2312.485223772" watchObservedRunningTime="2026-07-14 18:48:33.056520838 +0000 UTC m=+2312.488376272" Jul 14 18:48:33.127806 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:33.127770 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:33.132749 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:33.132721 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:34.037524 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:34.037490 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:34.038627 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:34.038608 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:47.106106 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:47.106067 2579 generic.go:356] "Generic (PLEG): container finished" podID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerID="a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12" exitCode=0 Jul 14 18:48:47.106616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:47.106147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" event={"ID":"426d9fb0-b6fa-4bc7-9774-60ea67aaf152","Type":"ContainerDied","Data":"a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12"} Jul 14 18:48:47.106616 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:47.106537 2579 scope.go:117] "RemoveContainer" containerID="a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12" Jul 14 18:48:47.107627 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:47.107598 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:48:47.107704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:47.107683 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:48:47.107741 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:47.107728 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:48:48.114855 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:48.114822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" event={"ID":"426d9fb0-b6fa-4bc7-9774-60ea67aaf152","Type":"ContainerStarted","Data":"bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8"} Jul 14 18:48:48.128150 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:48.128125 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:48.129191 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:48.129168 2579 patch_prober.go:28] interesting pod/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 18:48:48.129292 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:48.129220 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 18:48:49.037636 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:49.037591 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:49.127611 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:49.127577 2579 patch_prober.go:28] interesting pod/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 18:48:49.127987 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:49.127627 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 18:48:50.128195 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:50.128166 2579 patch_prober.go:28] interesting pod/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 18:48:50.128597 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:50.128210 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 18:48:51.128456 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:51.128422 2579 patch_prober.go:28] interesting pod/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 18:48:51.128862 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:51.128473 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 18:48:52.131484 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:52.131458 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:52.132724 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:52.132698 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:48:57.246861 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.246778 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94"] Jul 14 18:48:57.247322 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.247303 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" Jul 14 18:48:57.247451 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.247325 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b286750-826d-41b0-9c87-709dfaa6fb7a" containerName="istio-proxy" Jul 14 18:48:57.250946 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.250926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.254690 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.254667 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"openshift-service-ca.crt\"" Jul 14 18:48:57.254690 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.254686 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"auth-test-model-5bb958-kserve-self-signed-certs\"" Jul 14 18:48:57.254690 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.254670 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"kube-root-ca.crt\"" Jul 14 18:48:57.254911 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.254667 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"default-dockercfg-xfs96\"" Jul 14 18:48:57.267351 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.267318 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94"] Jul 14 18:48:57.394726 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.394688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhxk\" (UniqueName: \"kubernetes.io/projected/722dc869-8eaa-42e2-8226-e1d01401ae51-kube-api-access-4bhxk\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.394899 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.394747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722dc869-8eaa-42e2-8226-e1d01401ae51-tls-certs\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.394899 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.394806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-model-cache\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.394899 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.394890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-home\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.395009 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.394916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-dshm\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.395009 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.394950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-tmp-dir\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.495316 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-tmp-dir\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.495521 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhxk\" (UniqueName: \"kubernetes.io/projected/722dc869-8eaa-42e2-8226-e1d01401ae51-kube-api-access-4bhxk\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.495521 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722dc869-8eaa-42e2-8226-e1d01401ae51-tls-certs\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.495521 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-model-cache\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.495521 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-home\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.495521 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-dshm\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.496147 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-tmp-dir\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.496147 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495896 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-model-cache\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.496147 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.495946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-home\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.497833 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.497776 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-dshm\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.498280 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.498257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722dc869-8eaa-42e2-8226-e1d01401ae51-tls-certs\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.506042 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.506021 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhxk\" (UniqueName: \"kubernetes.io/projected/722dc869-8eaa-42e2-8226-e1d01401ae51-kube-api-access-4bhxk\") pod \"auth-test-model-5bb958-kserve-6cf469cd47-npl94\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.565273 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.565238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:48:57.700860 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:57.700836 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94"] Jul 14 18:48:57.703478 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:48:57.703455 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod722dc869_8eaa_42e2_8226_e1d01401ae51.slice/crio-cfee21db1da773816301cd96c291a3d375278b538ffceac4b72808da38c97ae6 WatchSource:0}: Error finding container cfee21db1da773816301cd96c291a3d375278b538ffceac4b72808da38c97ae6: Status 404 returned error can't find the container with id cfee21db1da773816301cd96c291a3d375278b538ffceac4b72808da38c97ae6 Jul 14 18:48:58.166007 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:58.165903 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" event={"ID":"722dc869-8eaa-42e2-8226-e1d01401ae51","Type":"ContainerStarted","Data":"ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d"} Jul 14 18:48:58.166007 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:58.165951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" event={"ID":"722dc869-8eaa-42e2-8226-e1d01401ae51","Type":"ContainerStarted","Data":"cfee21db1da773816301cd96c291a3d375278b538ffceac4b72808da38c97ae6"} Jul 14 18:48:58.193816 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:48:58.193763 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" podStartSLOduration=1.193745953 podStartE2EDuration="1.193745953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:48:58.191598845 +0000 UTC m=+2337.623454280" watchObservedRunningTime="2026-07-14 18:48:58.193745953 +0000 UTC m=+2337.625601387" Jul 14 18:49:07.566445 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:49:07.566404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:49:07.566936 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:49:07.566458 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:49:07.578913 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:49:07.578889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:49:08.229398 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:49:08.229367 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:50:01.254610 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:50:01.254583 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:50:01.260872 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:50:01.260850 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:51:22.457475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:22.457437 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94"] Jul 14 18:51:37.477489 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.477418 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" podUID="722dc869-8eaa-42e2-8226-e1d01401ae51" containerName="main" containerID="cri-o://ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d" gracePeriod=45 Jul 14 18:51:37.729190 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.729125 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:51:37.825196 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-model-cache\") pod \"722dc869-8eaa-42e2-8226-e1d01401ae51\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " Jul 14 18:51:37.825196 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825203 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722dc869-8eaa-42e2-8226-e1d01401ae51-tls-certs\") pod \"722dc869-8eaa-42e2-8226-e1d01401ae51\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " Jul 14 18:51:37.825457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825251 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-tmp-dir\") pod \"722dc869-8eaa-42e2-8226-e1d01401ae51\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " Jul 14 18:51:37.825457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825293 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-home\") pod \"722dc869-8eaa-42e2-8226-e1d01401ae51\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " Jul 14 18:51:37.825457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825322 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bhxk\" (UniqueName: \"kubernetes.io/projected/722dc869-8eaa-42e2-8226-e1d01401ae51-kube-api-access-4bhxk\") pod \"722dc869-8eaa-42e2-8226-e1d01401ae51\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " Jul 14 18:51:37.825457 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825374 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-dshm\") pod \"722dc869-8eaa-42e2-8226-e1d01401ae51\" (UID: \"722dc869-8eaa-42e2-8226-e1d01401ae51\") " Jul 14 18:51:37.825707 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825511 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-model-cache" (OuterVolumeSpecName: "model-cache") pod "722dc869-8eaa-42e2-8226-e1d01401ae51" (UID: "722dc869-8eaa-42e2-8226-e1d01401ae51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:51:37.825707 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825608 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "722dc869-8eaa-42e2-8226-e1d01401ae51" (UID: "722dc869-8eaa-42e2-8226-e1d01401ae51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:51:37.825707 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825657 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:51:37.825707 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.825689 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-home" (OuterVolumeSpecName: "home") pod "722dc869-8eaa-42e2-8226-e1d01401ae51" (UID: "722dc869-8eaa-42e2-8226-e1d01401ae51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:51:37.827742 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.827715 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722dc869-8eaa-42e2-8226-e1d01401ae51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "722dc869-8eaa-42e2-8226-e1d01401ae51" (UID: "722dc869-8eaa-42e2-8226-e1d01401ae51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:51:37.827841 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.827722 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-dshm" (OuterVolumeSpecName: "dshm") pod "722dc869-8eaa-42e2-8226-e1d01401ae51" (UID: "722dc869-8eaa-42e2-8226-e1d01401ae51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:51:37.827841 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.827787 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722dc869-8eaa-42e2-8226-e1d01401ae51-kube-api-access-4bhxk" (OuterVolumeSpecName: "kube-api-access-4bhxk") pod "722dc869-8eaa-42e2-8226-e1d01401ae51" (UID: "722dc869-8eaa-42e2-8226-e1d01401ae51"). InnerVolumeSpecName "kube-api-access-4bhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:51:37.926618 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.926545 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:51:37.926618 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.926612 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bhxk\" (UniqueName: \"kubernetes.io/projected/722dc869-8eaa-42e2-8226-e1d01401ae51-kube-api-access-4bhxk\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:51:37.926618 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.926627 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:51:37.926885 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.926640 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722dc869-8eaa-42e2-8226-e1d01401ae51-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:51:37.926885 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.926681 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722dc869-8eaa-42e2-8226-e1d01401ae51-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:51:37.994630 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.994512 2579 generic.go:356] "Generic (PLEG): container finished" podID="722dc869-8eaa-42e2-8226-e1d01401ae51" containerID="ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d" exitCode=0 Jul 14 18:51:37.994630 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.994576 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" Jul 14 18:51:37.994630 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.994600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" event={"ID":"722dc869-8eaa-42e2-8226-e1d01401ae51","Type":"ContainerDied","Data":"ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d"} Jul 14 18:51:37.994890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.994644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94" event={"ID":"722dc869-8eaa-42e2-8226-e1d01401ae51","Type":"ContainerDied","Data":"cfee21db1da773816301cd96c291a3d375278b538ffceac4b72808da38c97ae6"} Jul 14 18:51:37.994890 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:37.994661 2579 scope.go:117] "RemoveContainer" containerID="ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d" Jul 14 18:51:38.004289 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:38.004271 2579 scope.go:117] "RemoveContainer" containerID="ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d" Jul 14 18:51:38.004604 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:51:38.004580 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d\": container with ID starting with ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d not found: ID does not exist" containerID="ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d" Jul 14 18:51:38.004669 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:38.004614 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d"} err="failed to get container status \"ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d\": rpc error: code = NotFound desc = could not find container \"ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d\": container with ID starting with ede06ec9365d9afecbd120d8299d1ad801bc82abbb0c7cebe2146fcf5b69615d not found: ID does not exist" Jul 14 18:51:38.018990 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:38.018963 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94"] Jul 14 18:51:38.021735 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:38.021715 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/auth-test-model-5bb958-kserve-6cf469cd47-npl94"] Jul 14 18:51:39.207011 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:39.206981 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722dc869-8eaa-42e2-8226-e1d01401ae51" path="/var/lib/kubelet/pods/722dc869-8eaa-42e2-8226-e1d01401ae51/volumes" Jul 14 18:51:56.357273 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.357197 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz"] Jul 14 18:51:56.357740 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.357547 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722dc869-8eaa-42e2-8226-e1d01401ae51" containerName="main" Jul 14 18:51:56.357740 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.357575 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="722dc869-8eaa-42e2-8226-e1d01401ae51" containerName="main" Jul 14 18:51:56.357740 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.357636 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="722dc869-8eaa-42e2-8226-e1d01401ae51" containerName="main" Jul 14 18:51:56.360713 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.360695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.363823 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.363799 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"openshift-service-ca.crt\"" Jul 14 18:51:56.364955 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.364939 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"kube-root-ca.crt\"" Jul 14 18:51:56.365042 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.364944 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"default-dockercfg-q6qdm\"" Jul 14 18:51:56.365233 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.365213 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"sub-test-model-41a771-kserve-self-signed-certs\"" Jul 14 18:51:56.371109 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.371080 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz"] Jul 14 18:51:56.378251 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.378228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d1204e-1274-471c-ae72-646e3b53c8f3-tls-certs\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.378352 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.378259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-tmp-dir\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.378352 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.378283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-home\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.378352 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.378299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-model-cache\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.378483 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.378387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmn6l\" (UniqueName: \"kubernetes.io/projected/71d1204e-1274-471c-ae72-646e3b53c8f3-kube-api-access-tmn6l\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.378483 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.378464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-dshm\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.479687 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.479652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d1204e-1274-471c-ae72-646e3b53c8f3-tls-certs\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.479687 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.479689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-tmp-dir\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.479942 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.479710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-home\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.479942 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.479838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-model-cache\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.479942 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.479913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmn6l\" (UniqueName: \"kubernetes.io/projected/71d1204e-1274-471c-ae72-646e3b53c8f3-kube-api-access-tmn6l\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.480101 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.479992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-dshm\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.480101 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.480062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-tmp-dir\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.480198 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.480128 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-home\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.480234 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.480217 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-model-cache\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.482080 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.482058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-dshm\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.482308 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.482270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d1204e-1274-471c-ae72-646e3b53c8f3-tls-certs\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.492968 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.492942 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmn6l\" (UniqueName: \"kubernetes.io/projected/71d1204e-1274-471c-ae72-646e3b53c8f3-kube-api-access-tmn6l\") pod \"sub-test-model-41a771-kserve-b489fdf8f-jrtwz\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.670989 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.670903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:51:56.809724 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:56.809691 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz"] Jul 14 18:51:56.811385 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:51:56.811360 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71d1204e_1274_471c_ae72_646e3b53c8f3.slice/crio-764e5b543c4223380f93f03257e86e18ad12eb9424a28a1cdcae6709660c107f WatchSource:0}: Error finding container 764e5b543c4223380f93f03257e86e18ad12eb9424a28a1cdcae6709660c107f: Status 404 returned error can't find the container with id 764e5b543c4223380f93f03257e86e18ad12eb9424a28a1cdcae6709660c107f Jul 14 18:51:57.092725 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:57.092690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" event={"ID":"71d1204e-1274-471c-ae72-646e3b53c8f3","Type":"ContainerStarted","Data":"881a133f9fe620e05e6d7ac575c600822751c9627e21c1428b047adc442f0f90"} Jul 14 18:51:57.092725 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:57.092732 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" event={"ID":"71d1204e-1274-471c-ae72-646e3b53c8f3","Type":"ContainerStarted","Data":"764e5b543c4223380f93f03257e86e18ad12eb9424a28a1cdcae6709660c107f"} Jul 14 18:51:57.115241 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:51:57.115186 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" podStartSLOduration=1.115172154 podStartE2EDuration="1.115172154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:51:57.112788192 +0000 UTC m=+2516.544643625" watchObservedRunningTime="2026-07-14 18:51:57.115172154 +0000 UTC m=+2516.547027588" Jul 14 18:52:06.671965 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:52:06.671928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:52:06.672466 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:52:06.672012 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:52:06.684419 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:52:06.684392 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:52:07.161654 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:52:07.161618 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:52:52.240121 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:52:52.240073 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz"] Jul 14 18:53:07.260410 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.260346 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" podUID="71d1204e-1274-471c-ae72-646e3b53c8f3" containerName="main" containerID="cri-o://881a133f9fe620e05e6d7ac575c600822751c9627e21c1428b047adc442f0f90" gracePeriod=45 Jul 14 18:53:07.467356 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.467323 2579 generic.go:356] "Generic (PLEG): container finished" podID="71d1204e-1274-471c-ae72-646e3b53c8f3" containerID="881a133f9fe620e05e6d7ac575c600822751c9627e21c1428b047adc442f0f90" exitCode=0 Jul 14 18:53:07.467520 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.467392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" event={"ID":"71d1204e-1274-471c-ae72-646e3b53c8f3","Type":"ContainerDied","Data":"881a133f9fe620e05e6d7ac575c600822751c9627e21c1428b047adc442f0f90"} Jul 14 18:53:07.519027 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.518945 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:53:07.625084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625049 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d1204e-1274-471c-ae72-646e3b53c8f3-tls-certs\") pod \"71d1204e-1274-471c-ae72-646e3b53c8f3\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " Jul 14 18:53:07.625084 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625084 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-dshm\") pod \"71d1204e-1274-471c-ae72-646e3b53c8f3\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " Jul 14 18:53:07.625347 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625121 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-tmp-dir\") pod \"71d1204e-1274-471c-ae72-646e3b53c8f3\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " Jul 14 18:53:07.625347 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625137 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-model-cache\") pod \"71d1204e-1274-471c-ae72-646e3b53c8f3\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " Jul 14 18:53:07.625347 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625182 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-home\") pod \"71d1204e-1274-471c-ae72-646e3b53c8f3\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " Jul 14 18:53:07.625347 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmn6l\" (UniqueName: \"kubernetes.io/projected/71d1204e-1274-471c-ae72-646e3b53c8f3-kube-api-access-tmn6l\") pod \"71d1204e-1274-471c-ae72-646e3b53c8f3\" (UID: \"71d1204e-1274-471c-ae72-646e3b53c8f3\") " Jul 14 18:53:07.625600 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625421 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71d1204e-1274-471c-ae72-646e3b53c8f3" (UID: "71d1204e-1274-471c-ae72-646e3b53c8f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:07.625600 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "71d1204e-1274-471c-ae72-646e3b53c8f3" (UID: "71d1204e-1274-471c-ae72-646e3b53c8f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:07.625600 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.625511 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-home" (OuterVolumeSpecName: "home") pod "71d1204e-1274-471c-ae72-646e3b53c8f3" (UID: "71d1204e-1274-471c-ae72-646e3b53c8f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:07.627412 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.627390 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-dshm" (OuterVolumeSpecName: "dshm") pod "71d1204e-1274-471c-ae72-646e3b53c8f3" (UID: "71d1204e-1274-471c-ae72-646e3b53c8f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:07.627412 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.627393 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d1204e-1274-471c-ae72-646e3b53c8f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71d1204e-1274-471c-ae72-646e3b53c8f3" (UID: "71d1204e-1274-471c-ae72-646e3b53c8f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:53:07.627546 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.627505 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d1204e-1274-471c-ae72-646e3b53c8f3-kube-api-access-tmn6l" (OuterVolumeSpecName: "kube-api-access-tmn6l") pod "71d1204e-1274-471c-ae72-646e3b53c8f3" (UID: "71d1204e-1274-471c-ae72-646e3b53c8f3"). InnerVolumeSpecName "kube-api-access-tmn6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:53:07.726639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.726595 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:53:07.726639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.726631 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:53:07.726639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.726640 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:53:07.726639 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.726648 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmn6l\" (UniqueName: \"kubernetes.io/projected/71d1204e-1274-471c-ae72-646e3b53c8f3-kube-api-access-tmn6l\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:53:07.726919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.726656 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d1204e-1274-471c-ae72-646e3b53c8f3-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:53:07.726919 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:07.726666 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d1204e-1274-471c-ae72-646e3b53c8f3-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:53:08.473624 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:08.473591 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" Jul 14 18:53:08.473624 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:08.473608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz" event={"ID":"71d1204e-1274-471c-ae72-646e3b53c8f3","Type":"ContainerDied","Data":"764e5b543c4223380f93f03257e86e18ad12eb9424a28a1cdcae6709660c107f"} Jul 14 18:53:08.474140 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:08.473657 2579 scope.go:117] "RemoveContainer" containerID="881a133f9fe620e05e6d7ac575c600822751c9627e21c1428b047adc442f0f90" Jul 14 18:53:08.498080 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:08.498050 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz"] Jul 14 18:53:08.502920 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:08.502893 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/sub-test-model-41a771-kserve-b489fdf8f-jrtwz"] Jul 14 18:53:09.207702 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:09.207672 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71d1204e-1274-471c-ae72-646e3b53c8f3" path="/var/lib/kubelet/pods/71d1204e-1274-471c-ae72-646e3b53c8f3/volumes" Jul 14 18:53:32.579131 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.578912 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw"] Jul 14 18:53:32.579717 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.579630 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d1204e-1274-471c-ae72-646e3b53c8f3" containerName="main" Jul 14 18:53:32.579717 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.579651 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d1204e-1274-471c-ae72-646e3b53c8f3" containerName="main" Jul 14 18:53:32.579839 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.579828 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d1204e-1274-471c-ae72-646e3b53c8f3" containerName="main" Jul 14 18:53:32.583959 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.583935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.586895 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.586867 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"openshift-service-ca.crt\"" Jul 14 18:53:32.587022 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.586951 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"kube-root-ca.crt\"" Jul 14 18:53:32.588113 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.588093 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"default-dockercfg-q6qdm\"" Jul 14 18:53:32.588253 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.588135 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"rate-test-model-41a771-kserve-self-signed-certs\"" Jul 14 18:53:32.607346 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.607312 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw"] Jul 14 18:53:32.649882 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.649847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwwk\" (UniqueName: \"kubernetes.io/projected/389773cc-049d-46d8-9017-f3f8e3905d6c-kube-api-access-hqwwk\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.649882 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.649885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-home\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.650092 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.649911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-tmp-dir\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.650092 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.649962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/389773cc-049d-46d8-9017-f3f8e3905d6c-tls-certs\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.650092 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.649984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-dshm\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.650092 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.650054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-model-cache\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.750684 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.750649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-model-cache\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.750928 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.750723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwwk\" (UniqueName: \"kubernetes.io/projected/389773cc-049d-46d8-9017-f3f8e3905d6c-kube-api-access-hqwwk\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.750928 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.750745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-home\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.750928 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.750769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-tmp-dir\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.750928 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.750789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/389773cc-049d-46d8-9017-f3f8e3905d6c-tls-certs\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.750928 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.750805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-dshm\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.751220 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.751103 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-model-cache\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.751220 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.751160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-home\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.751330 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.751276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-tmp-dir\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.753104 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.753079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-dshm\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.753397 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.753379 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/389773cc-049d-46d8-9017-f3f8e3905d6c-tls-certs\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.765638 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.765609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwwk\" (UniqueName: \"kubernetes.io/projected/389773cc-049d-46d8-9017-f3f8e3905d6c-kube-api-access-hqwwk\") pod \"rate-test-model-41a771-kserve-7494496595-9kxbw\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:32.892715 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:32.892621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:33.029998 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:33.029967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw"] Jul 14 18:53:33.030984 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:53:33.030953 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod389773cc_049d_46d8_9017_f3f8e3905d6c.slice/crio-a3d9805ffee54c0e6335627572a275594813d268a71bef8cbdf1f8b050b574c3 WatchSource:0}: Error finding container a3d9805ffee54c0e6335627572a275594813d268a71bef8cbdf1f8b050b574c3: Status 404 returned error can't find the container with id a3d9805ffee54c0e6335627572a275594813d268a71bef8cbdf1f8b050b574c3 Jul 14 18:53:33.603984 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:33.603949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" event={"ID":"389773cc-049d-46d8-9017-f3f8e3905d6c","Type":"ContainerStarted","Data":"d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33"} Jul 14 18:53:33.603984 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:33.603990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" event={"ID":"389773cc-049d-46d8-9017-f3f8e3905d6c","Type":"ContainerStarted","Data":"a3d9805ffee54c0e6335627572a275594813d268a71bef8cbdf1f8b050b574c3"} Jul 14 18:53:33.629342 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:33.629290 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" podStartSLOduration=1.629274441 podStartE2EDuration="1.629274441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:53:33.628886581 +0000 UTC m=+2613.060742016" watchObservedRunningTime="2026-07-14 18:53:33.629274441 +0000 UTC m=+2613.061129876" Jul 14 18:53:42.893631 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:42.893590 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:42.894157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:42.893666 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:42.906321 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:42.906295 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:53:43.666169 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:53:43.666140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:54:16.171300 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:16.171263 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw"] Jul 14 18:54:31.191249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.191196 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" podUID="389773cc-049d-46d8-9017-f3f8e3905d6c" containerName="main" containerID="cri-o://d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33" gracePeriod=45 Jul 14 18:54:31.445368 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.445307 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:54:31.459194 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqwwk\" (UniqueName: \"kubernetes.io/projected/389773cc-049d-46d8-9017-f3f8e3905d6c-kube-api-access-hqwwk\") pod \"389773cc-049d-46d8-9017-f3f8e3905d6c\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " Jul 14 18:54:31.459313 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459217 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-model-cache\") pod \"389773cc-049d-46d8-9017-f3f8e3905d6c\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " Jul 14 18:54:31.459313 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459256 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-home\") pod \"389773cc-049d-46d8-9017-f3f8e3905d6c\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " Jul 14 18:54:31.459313 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459308 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/389773cc-049d-46d8-9017-f3f8e3905d6c-tls-certs\") pod \"389773cc-049d-46d8-9017-f3f8e3905d6c\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " Jul 14 18:54:31.459432 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459354 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-dshm\") pod \"389773cc-049d-46d8-9017-f3f8e3905d6c\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " Jul 14 18:54:31.459432 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-tmp-dir\") pod \"389773cc-049d-46d8-9017-f3f8e3905d6c\" (UID: \"389773cc-049d-46d8-9017-f3f8e3905d6c\") " Jul 14 18:54:31.459607 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459525 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-home" (OuterVolumeSpecName: "home") pod "389773cc-049d-46d8-9017-f3f8e3905d6c" (UID: "389773cc-049d-46d8-9017-f3f8e3905d6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:54:31.459607 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459538 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "389773cc-049d-46d8-9017-f3f8e3905d6c" (UID: "389773cc-049d-46d8-9017-f3f8e3905d6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:54:31.459817 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459696 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:54:31.459817 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459710 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:54:31.459817 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.459730 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "389773cc-049d-46d8-9017-f3f8e3905d6c" (UID: "389773cc-049d-46d8-9017-f3f8e3905d6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:54:31.461339 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.461314 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389773cc-049d-46d8-9017-f3f8e3905d6c-kube-api-access-hqwwk" (OuterVolumeSpecName: "kube-api-access-hqwwk") pod "389773cc-049d-46d8-9017-f3f8e3905d6c" (UID: "389773cc-049d-46d8-9017-f3f8e3905d6c"). InnerVolumeSpecName "kube-api-access-hqwwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:31.461473 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.461452 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389773cc-049d-46d8-9017-f3f8e3905d6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "389773cc-049d-46d8-9017-f3f8e3905d6c" (UID: "389773cc-049d-46d8-9017-f3f8e3905d6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:54:31.461533 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.461478 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-dshm" (OuterVolumeSpecName: "dshm") pod "389773cc-049d-46d8-9017-f3f8e3905d6c" (UID: "389773cc-049d-46d8-9017-f3f8e3905d6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:54:31.560828 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.560790 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/389773cc-049d-46d8-9017-f3f8e3905d6c-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:54:31.560828 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.560821 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:54:31.560828 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.560829 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/389773cc-049d-46d8-9017-f3f8e3905d6c-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:54:31.561077 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.560840 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqwwk\" (UniqueName: \"kubernetes.io/projected/389773cc-049d-46d8-9017-f3f8e3905d6c-kube-api-access-hqwwk\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:54:31.909100 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.909062 2579 generic.go:356] "Generic (PLEG): container finished" podID="389773cc-049d-46d8-9017-f3f8e3905d6c" containerID="d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33" exitCode=0 Jul 14 18:54:31.909283 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.909112 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" Jul 14 18:54:31.909283 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.909146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" event={"ID":"389773cc-049d-46d8-9017-f3f8e3905d6c","Type":"ContainerDied","Data":"d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33"} Jul 14 18:54:31.909283 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.909193 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw" event={"ID":"389773cc-049d-46d8-9017-f3f8e3905d6c","Type":"ContainerDied","Data":"a3d9805ffee54c0e6335627572a275594813d268a71bef8cbdf1f8b050b574c3"} Jul 14 18:54:31.909283 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.909211 2579 scope.go:117] "RemoveContainer" containerID="d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33" Jul 14 18:54:31.919235 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.919218 2579 scope.go:117] "RemoveContainer" containerID="d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33" Jul 14 18:54:31.919533 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:54:31.919508 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33\": container with ID starting with d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33 not found: ID does not exist" containerID="d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33" Jul 14 18:54:31.919640 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.919539 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33"} err="failed to get container status \"d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33\": rpc error: code = NotFound desc = could not find container \"d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33\": container with ID starting with d7b41d7218635c175c4ca83694fdb8751d7882a7ba61e3a3e3d1614c9f73dc33 not found: ID does not exist" Jul 14 18:54:31.933669 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.933642 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw"] Jul 14 18:54:31.938341 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:31.938318 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/rate-test-model-41a771-kserve-7494496595-9kxbw"] Jul 14 18:54:33.207499 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:54:33.207464 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="389773cc-049d-46d8-9017-f3f8e3905d6c" path="/var/lib/kubelet/pods/389773cc-049d-46d8-9017-f3f8e3905d6c/volumes" Jul 14 18:55:01.286657 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:01.286581 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:55:01.299675 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:01.299652 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z5gxk_4f4f89fd-4d5d-4944-8c8f-2c1d16d46ef3/ovn-acl-logging/0.log" Jul 14 18:55:27.862336 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:27.862303 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44"] Jul 14 18:55:27.862785 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:27.862548 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" containerID="cri-o://bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8" gracePeriod=30 Jul 14 18:55:28.656735 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:28.656704 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9c594c5-v45s2_effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc/authorino/0.log" Jul 14 18:55:33.120419 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.120395 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:55:33.177492 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-socket\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.177492 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177466 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-certs\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.177772 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177496 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istiod-ca-cert\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.177772 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177524 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-token\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.177772 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-envoy\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.177772 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177619 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-credential-socket\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.177772 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177648 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-podinfo\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.177772 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177674 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84bvz\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-kube-api-access-84bvz\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.177772 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177696 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-data\") pod \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\" (UID: \"426d9fb0-b6fa-4bc7-9774-60ea67aaf152\") " Jul 14 18:55:33.178139 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177770 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:33.178139 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177909 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:33.178139 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.177993 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:33.178139 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.178018 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:55:33.178344 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.178224 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:33.178724 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.178696 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-data" (OuterVolumeSpecName: "istio-data") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:33.179876 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.179850 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-kube-api-access-84bvz" (OuterVolumeSpecName: "kube-api-access-84bvz") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "kube-api-access-84bvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:55:33.179988 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.179921 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-token" (OuterVolumeSpecName: "istio-token") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:55:33.179988 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.179939 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:55:33.179988 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.179955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "426d9fb0-b6fa-4bc7-9774-60ea67aaf152" (UID: "426d9fb0-b6fa-4bc7-9774-60ea67aaf152"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:33.235195 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.235162 2579 generic.go:356] "Generic (PLEG): container finished" podID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerID="bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8" exitCode=0 Jul 14 18:55:33.235370 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.235218 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" Jul 14 18:55:33.235370 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.235254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" event={"ID":"426d9fb0-b6fa-4bc7-9774-60ea67aaf152","Type":"ContainerDied","Data":"bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8"} Jul 14 18:55:33.235370 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.235325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44" event={"ID":"426d9fb0-b6fa-4bc7-9774-60ea67aaf152","Type":"ContainerDied","Data":"088fda1664fffc75895f90d3c0f56372cefcdbfab6db2eae829034ba05a07a0e"} Jul 14 18:55:33.235370 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.235354 2579 scope.go:117] "RemoveContainer" containerID="bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8" Jul 14 18:55:33.244188 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.244169 2579 scope.go:117] "RemoveContainer" containerID="a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12" Jul 14 18:55:33.257301 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.257268 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44"] Jul 14 18:55:33.257995 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.257978 2579 scope.go:117] "RemoveContainer" containerID="bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8" Jul 14 18:55:33.258303 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:55:33.258283 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8\": container with ID starting with bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8 not found: ID does not exist" containerID="bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8" Jul 14 18:55:33.258350 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.258313 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8"} err="failed to get container status \"bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8\": rpc error: code = NotFound desc = could not find container \"bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8\": container with ID starting with bbb9165dc58e81bb81ae47e77bd0ebe3195fe61290f8b6c91132551aaea4faa8 not found: ID does not exist" Jul 14 18:55:33.258350 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.258333 2579 scope.go:117] "RemoveContainer" containerID="a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12" Jul 14 18:55:33.258602 ip-10-0-129-146 kubenswrapper[2579]: E0714 18:55:33.258579 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12\": container with ID starting with a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12 not found: ID does not exist" containerID="a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12" Jul 14 18:55:33.258652 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.258609 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12"} err="failed to get container status \"a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12\": rpc error: code = NotFound desc = could not find container \"a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12\": container with ID starting with a0c32ebe393759580dd632421de0922649c0faf56e547470a673f04cc1b43c12 not found: ID does not exist" Jul 14 18:55:33.265469 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.265445 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-41a771-openshift-default-6886556bcf-86f44"] Jul 14 18:55:33.279298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.279254 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:33.279298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.279292 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:33.279298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.279300 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:33.279298 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.279308 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:33.279532 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.279316 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:33.279532 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.279325 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:33.279532 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.279333 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84bvz\" (UniqueName: \"kubernetes.io/projected/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-kube-api-access-84bvz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:33.279532 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:33.279341 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d9fb0-b6fa-4bc7-9774-60ea67aaf152-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\"" Jul 14 18:55:34.465050 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:34.465019 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lzz2q_b9f3c473-0e59-40bd-a209-9bd8e76b000f/server/0.log" Jul 14 18:55:35.027844 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:35.027817 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-rsnn4_b62b7fd6-cbb3-4fc4-acad-feee534701bc/copy-manifests/0.log" Jul 14 18:55:35.038691 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:35.038668 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-rsnn4_b62b7fd6-cbb3-4fc4-acad-feee534701bc/manager/0.log" Jul 14 18:55:35.207767 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:35.207734 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" path="/var/lib/kubelet/pods/426d9fb0-b6fa-4bc7-9774-60ea67aaf152/volumes" Jul 14 18:55:38.177758 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.177723 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn_b1c1c286-bf1f-45ac-af37-d49925719e3a/pull/0.log" Jul 14 18:55:38.186282 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.186259 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn_b1c1c286-bf1f-45ac-af37-d49925719e3a/extract/0.log" Jul 14 18:55:38.194291 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.194270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn_b1c1c286-bf1f-45ac-af37-d49925719e3a/util/0.log" Jul 14 18:55:38.319811 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.319783 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl_3d8796e6-2680-4c22-8b6d-4c77588c7676/extract/0.log" Jul 14 18:55:38.327757 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.327731 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl_3d8796e6-2680-4c22-8b6d-4c77588c7676/util/0.log" Jul 14 18:55:38.336116 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.336094 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl_3d8796e6-2680-4c22-8b6d-4c77588c7676/pull/0.log" Jul 14 18:55:38.460111 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.460082 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q_6a93c87e-3d84-4384-a32c-84e87fd39d42/pull/0.log" Jul 14 18:55:38.469064 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.469045 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q_6a93c87e-3d84-4384-a32c-84e87fd39d42/extract/0.log" Jul 14 18:55:38.477699 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.477680 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q_6a93c87e-3d84-4384-a32c-84e87fd39d42/util/0.log" Jul 14 18:55:38.606388 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.606362 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7_ef1c96dc-c802-40b9-933e-173d06bf9a13/util/0.log" Jul 14 18:55:38.614573 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.614536 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7_ef1c96dc-c802-40b9-933e-173d06bf9a13/pull/0.log" Jul 14 18:55:38.622042 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.622024 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7_ef1c96dc-c802-40b9-933e-173d06bf9a13/extract/0.log" Jul 14 18:55:38.750831 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.750736 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9c594c5-v45s2_effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc/authorino/0.log" Jul 14 18:55:38.888159 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:38.888130 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-j9x7q_6099eeb4-692c-4ee7-8908-6307b56aa223/manager/0.log" Jul 14 18:55:39.014083 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:39.014012 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-j4pfl_d48b7991-d77c-4705-96af-9754fadfebf3/manager/0.log" Jul 14 18:55:39.133795 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:39.133767 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9gdvn_7a4b8ab6-5d07-489d-96a7-c255b9115400/kuadrant-console-plugin/0.log" Jul 14 18:55:39.259151 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:39.259124 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2q25z_d42ca5a6-1f6f-4f17-8394-75660bcd6a2d/registry-server/0.log" Jul 14 18:55:39.640275 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:39.640243 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tpxq7_62bc3629-2691-4994-890b-95ce711820da/manager/0.log" Jul 14 18:55:40.027917 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:40.027891 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6785s5b9_b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4/istio-proxy/0.log" Jul 14 18:55:41.431748 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:41.431701 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-pcqj7_93aaa300-e330-45e3-a79d-a30df5e60540/main/0.log" Jul 14 18:55:41.803742 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:41.803715 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zb2ws_207dc962-216f-4514-b67e-84fb2ea9d460/main/0.log" Jul 14 18:55:41.931217 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:41.931191 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-7bx7r_816e572a-53b3-471f-bdc1-597c402a9804/main/0.log" Jul 14 18:55:47.071790 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.071752 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-492wj/must-gather-n6qs2"] Jul 14 18:55:47.072310 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072223 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" Jul 14 18:55:47.072310 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072245 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" Jul 14 18:55:47.072310 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072260 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" Jul 14 18:55:47.072310 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072267 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" Jul 14 18:55:47.072310 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072284 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="389773cc-049d-46d8-9017-f3f8e3905d6c" containerName="main" Jul 14 18:55:47.072310 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072293 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="389773cc-049d-46d8-9017-f3f8e3905d6c" containerName="main" Jul 14 18:55:47.072645 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072375 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" Jul 14 18:55:47.072645 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072389 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="426d9fb0-b6fa-4bc7-9774-60ea67aaf152" containerName="istio-proxy" Jul 14 18:55:47.072645 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.072399 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="389773cc-049d-46d8-9017-f3f8e3905d6c" containerName="main" Jul 14 18:55:47.075539 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.075519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-492wj/must-gather-n6qs2" Jul 14 18:55:47.078253 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.078232 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-492wj\"/\"openshift-service-ca.crt\"" Jul 14 18:55:47.079355 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.079332 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-492wj\"/\"kube-root-ca.crt\"" Jul 14 18:55:47.079454 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.079363 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-492wj\"/\"default-dockercfg-n6svq\"" Jul 14 18:55:47.084530 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.084508 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-492wj/must-gather-n6qs2"] Jul 14 18:55:47.204257 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.204221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2911bc05-e7d9-4dc6-88d1-befd8da1ac87-must-gather-output\") pod \"must-gather-n6qs2\" (UID: \"2911bc05-e7d9-4dc6-88d1-befd8da1ac87\") " pod="openshift-must-gather-492wj/must-gather-n6qs2" Jul 14 18:55:47.204427 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.204327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-865hz\" (UniqueName: \"kubernetes.io/projected/2911bc05-e7d9-4dc6-88d1-befd8da1ac87-kube-api-access-865hz\") pod \"must-gather-n6qs2\" (UID: \"2911bc05-e7d9-4dc6-88d1-befd8da1ac87\") " pod="openshift-must-gather-492wj/must-gather-n6qs2" Jul 14 18:55:47.305486 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.305447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2911bc05-e7d9-4dc6-88d1-befd8da1ac87-must-gather-output\") pod \"must-gather-n6qs2\" (UID: \"2911bc05-e7d9-4dc6-88d1-befd8da1ac87\") " pod="openshift-must-gather-492wj/must-gather-n6qs2" Jul 14 18:55:47.305712 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.305584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865hz\" (UniqueName: \"kubernetes.io/projected/2911bc05-e7d9-4dc6-88d1-befd8da1ac87-kube-api-access-865hz\") pod \"must-gather-n6qs2\" (UID: \"2911bc05-e7d9-4dc6-88d1-befd8da1ac87\") " pod="openshift-must-gather-492wj/must-gather-n6qs2" Jul 14 18:55:47.305838 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.305821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2911bc05-e7d9-4dc6-88d1-befd8da1ac87-must-gather-output\") pod \"must-gather-n6qs2\" (UID: \"2911bc05-e7d9-4dc6-88d1-befd8da1ac87\") " pod="openshift-must-gather-492wj/must-gather-n6qs2" Jul 14 18:55:47.315232 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.315204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-865hz\" (UniqueName: \"kubernetes.io/projected/2911bc05-e7d9-4dc6-88d1-befd8da1ac87-kube-api-access-865hz\") pod \"must-gather-n6qs2\" (UID: \"2911bc05-e7d9-4dc6-88d1-befd8da1ac87\") " pod="openshift-must-gather-492wj/must-gather-n6qs2" Jul 14 18:55:47.385160 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.385063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-492wj/must-gather-n6qs2" Jul 14 18:55:47.522307 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.522278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-492wj/must-gather-n6qs2"] Jul 14 18:55:47.524425 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:55:47.524403 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2911bc05_e7d9_4dc6_88d1_befd8da1ac87.slice/crio-b4f101a59b7a4f37adaa57ab2bef437950dc6d91999cec23c0dc9fe2f67a2ded WatchSource:0}: Error finding container b4f101a59b7a4f37adaa57ab2bef437950dc6d91999cec23c0dc9fe2f67a2ded: Status 404 returned error can't find the container with id b4f101a59b7a4f37adaa57ab2bef437950dc6d91999cec23c0dc9fe2f67a2ded Jul 14 18:55:47.526157 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:47.526140 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:55:48.313704 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:48.313677 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/must-gather-n6qs2" event={"ID":"2911bc05-e7d9-4dc6-88d1-befd8da1ac87","Type":"ContainerStarted","Data":"b4f101a59b7a4f37adaa57ab2bef437950dc6d91999cec23c0dc9fe2f67a2ded"} Jul 14 18:55:49.321975 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:49.321940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/must-gather-n6qs2" event={"ID":"2911bc05-e7d9-4dc6-88d1-befd8da1ac87","Type":"ContainerStarted","Data":"70509f22079b3d84db3472cd1df9bd4b4734754fef2bc4254b31694913f5a291"} Jul 14 18:55:49.322335 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:49.321984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/must-gather-n6qs2" event={"ID":"2911bc05-e7d9-4dc6-88d1-befd8da1ac87","Type":"ContainerStarted","Data":"5a217b969647401d7269da29e584d8114fb000c1ee07feaffde2afff0c38fe58"} Jul 14 18:55:49.341927 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:49.341864 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-492wj/must-gather-n6qs2" podStartSLOduration=1.603849763 podStartE2EDuration="2.341844384s" totalImagesPullingTime="737.994621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:49.340499718 +0000 UTC m=+2748.772355151" watchObservedRunningTime="2026-07-14 18:55:49.341844384 +0000 UTC m=+2748.773699819" Jul 14 18:55:50.032270 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:50.032239 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-82p7g_78631d62-ebdf-43af-a039-83ba0683e841/global-pull-secret-syncer/0.log" Jul 14 18:55:50.187581 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:50.187535 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j78bw_e6325f19-b72d-491c-9eb6-69569b111a9f/konnectivity-agent/0.log" Jul 14 18:55:50.251031 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:50.250997 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-146.ec2.internal_fd0eec76c28a01b9629dd9799f6aa948/haproxy/0.log" Jul 14 18:55:54.738083 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:54.738054 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn_b1c1c286-bf1f-45ac-af37-d49925719e3a/extract/0.log" Jul 14 18:55:54.778764 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:54.778725 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn_b1c1c286-bf1f-45ac-af37-d49925719e3a/util/0.log" Jul 14 18:55:54.813859 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:54.813831 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594svdn_b1c1c286-bf1f-45ac-af37-d49925719e3a/pull/0.log" Jul 14 18:55:54.852910 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:54.852867 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl_3d8796e6-2680-4c22-8b6d-4c77588c7676/extract/0.log" Jul 14 18:55:54.881335 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:54.881309 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl_3d8796e6-2680-4c22-8b6d-4c77588c7676/util/0.log" Jul 14 18:55:54.912736 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:54.912678 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e072bcl_3d8796e6-2680-4c22-8b6d-4c77588c7676/pull/0.log" Jul 14 18:55:54.951230 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:54.951203 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q_6a93c87e-3d84-4384-a32c-84e87fd39d42/extract/0.log" Jul 14 18:55:54.985321 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:54.985295 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q_6a93c87e-3d84-4384-a32c-84e87fd39d42/util/0.log" Jul 14 18:55:55.022901 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.022788 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n788q_6a93c87e-3d84-4384-a32c-84e87fd39d42/pull/0.log" Jul 14 18:55:55.066893 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.066854 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7_ef1c96dc-c802-40b9-933e-173d06bf9a13/extract/0.log" Jul 14 18:55:55.099534 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.099509 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7_ef1c96dc-c802-40b9-933e-173d06bf9a13/util/0.log" Jul 14 18:55:55.136607 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.136582 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t5rb7_ef1c96dc-c802-40b9-933e-173d06bf9a13/pull/0.log" Jul 14 18:55:55.586109 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.586071 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9c594c5-v45s2_effa659c-c9eb-4e9f-8bf5-2d12d1bcabcc/authorino/0.log" Jul 14 18:55:55.624615 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.624585 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-j9x7q_6099eeb4-692c-4ee7-8908-6307b56aa223/manager/0.log" Jul 14 18:55:55.654633 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.654565 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-j4pfl_d48b7991-d77c-4705-96af-9754fadfebf3/manager/0.log" Jul 14 18:55:55.696433 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.696395 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9gdvn_7a4b8ab6-5d07-489d-96a7-c255b9115400/kuadrant-console-plugin/0.log" Jul 14 18:55:55.753673 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:55.753589 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2q25z_d42ca5a6-1f6f-4f17-8394-75660bcd6a2d/registry-server/0.log" Jul 14 18:55:56.013755 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:56.013720 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tpxq7_62bc3629-2691-4994-890b-95ce711820da/manager/0.log" Jul 14 18:55:58.060660 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:58.060629 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdfnk_48386c71-621c-42dd-b9f9-5be94e9f50ad/node-exporter/0.log" Jul 14 18:55:58.087222 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:58.087195 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdfnk_48386c71-621c-42dd-b9f9-5be94e9f50ad/kube-rbac-proxy/0.log" Jul 14 18:55:58.114328 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:58.114300 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdfnk_48386c71-621c-42dd-b9f9-5be94e9f50ad/init-textfile/0.log" Jul 14 18:55:58.663567 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:58.663521 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-flnzr_78661dc5-00ea-49ca-866c-4e82cb3eead9/prometheus-operator-admission-webhook/0.log" Jul 14 18:55:59.715658 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.715626 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz"] Jul 14 18:55:59.723622 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.723595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.730612 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.730584 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz"] Jul 14 18:55:59.830161 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.830120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-podres\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.830353 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.830190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-sys\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.830353 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.830244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-proc\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.830353 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.830297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-lib-modules\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.830353 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.830325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4l58\" (UniqueName: \"kubernetes.io/projected/7319037f-1b21-4cd1-87da-b952e78b3196-kube-api-access-w4l58\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.930916 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.930881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-sys\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.931097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.930932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-proc\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.931097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.930976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-lib-modules\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.931097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.931001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4l58\" (UniqueName: \"kubernetes.io/projected/7319037f-1b21-4cd1-87da-b952e78b3196-kube-api-access-w4l58\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.931097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.931017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-sys\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.931097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.931054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-podres\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.931097 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.931083 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-proc\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.931346 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.931159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-podres\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.931346 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.931179 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7319037f-1b21-4cd1-87da-b952e78b3196-lib-modules\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:55:59.941657 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:55:59.941616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4l58\" (UniqueName: \"kubernetes.io/projected/7319037f-1b21-4cd1-87da-b952e78b3196-kube-api-access-w4l58\") pod \"perf-node-gather-daemonset-n4szz\" (UID: \"7319037f-1b21-4cd1-87da-b952e78b3196\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:56:00.037708 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:00.037610 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:56:00.224145 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:00.222794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz"] Jul 14 18:56:00.224810 ip-10-0-129-146 kubenswrapper[2579]: W0714 18:56:00.224766 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7319037f_1b21_4cd1_87da_b952e78b3196.slice/crio-9954942a438a25ef7a1e95b7acf787157624ee93f1b3d5d509bfeb9d782465ab WatchSource:0}: Error finding container 9954942a438a25ef7a1e95b7acf787157624ee93f1b3d5d509bfeb9d782465ab: Status 404 returned error can't find the container with id 9954942a438a25ef7a1e95b7acf787157624ee93f1b3d5d509bfeb9d782465ab Jul 14 18:56:00.423256 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:00.423212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" event={"ID":"7319037f-1b21-4cd1-87da-b952e78b3196","Type":"ContainerStarted","Data":"8479fe74d1e0370fae03ca8a07fc0c971ab247911be67c5b3f917b5e7597f677"} Jul 14 18:56:00.423446 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:00.423284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" event={"ID":"7319037f-1b21-4cd1-87da-b952e78b3196","Type":"ContainerStarted","Data":"9954942a438a25ef7a1e95b7acf787157624ee93f1b3d5d509bfeb9d782465ab"} Jul 14 18:56:00.423708 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:00.423669 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:56:01.355184 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:01.355145 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-wfqbp_e9024dcf-185d-412a-8d84-42605eca593a/download-server/0.log" Jul 14 18:56:03.223249 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:03.223216 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lg979_8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6/dns/0.log" Jul 14 18:56:03.249567 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:03.249516 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lg979_8ee0161c-9fa5-47f2-bbc2-b89afc69f8c6/kube-rbac-proxy/0.log" Jul 14 18:56:03.277029 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:03.277003 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84xwb_9c17b1f7-00f3-4d65-941f-16eb0c2e9d24/dns-node-resolver/0.log" Jul 14 18:56:03.909155 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:03.909125 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65f97494db-wtzsp_4e316ae5-cb89-435b-90d9-1e366152d608/registry/0.log" Jul 14 18:56:03.936475 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:03.936450 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cckpq_75b932e3-25bd-4bb8-a092-4de99b1b0625/node-ca/0.log" Jul 14 18:56:05.064311 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:05.064283 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6785s5b9_b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4/istio-proxy/0.log" Jul 14 18:56:06.182636 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:06.182611 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxkkc_88d206d5-a34a-4da9-8d97-2ccb5c1c707d/serve-healthcheck-canary/0.log" Jul 14 18:56:06.446115 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:06.445099 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" Jul 14 18:56:06.468364 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:06.467718 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-n4szz" podStartSLOduration=7.467694996 podStartE2EDuration="7.467694996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:56:00.444578057 +0000 UTC m=+2759.876433485" watchObservedRunningTime="2026-07-14 18:56:06.467694996 +0000 UTC m=+2765.899550431" Jul 14 18:56:06.818424 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:06.818398 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k7vdr_7ee50913-fbec-42f0-a1e8-55242e9c5ff7/kube-rbac-proxy/0.log" Jul 14 18:56:06.844260 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:06.844229 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k7vdr_7ee50913-fbec-42f0-a1e8-55242e9c5ff7/exporter/0.log" Jul 14 18:56:06.872320 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:06.872295 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k7vdr_7ee50913-fbec-42f0-a1e8-55242e9c5ff7/extractor/0.log" Jul 14 18:56:09.674950 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:09.674922 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lzz2q_b9f3c473-0e59-40bd-a209-9bd8e76b000f/server/0.log" Jul 14 18:56:09.934206 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:09.934120 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-rsnn4_b62b7fd6-cbb3-4fc4-acad-feee534701bc/manager/0.log" Jul 14 18:56:09.960189 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:09.960160 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-rsnn4_b62b7fd6-cbb3-4fc4-acad-feee534701bc/copy-manifests/0.log" Jul 14 18:56:10.318171 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:10.318139 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6785s5b9_b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4/istio-proxy/0.log" Jul 14 18:56:11.301956 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:11.301930 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-55b7dd5b8c-bl547_5c80640c-f601-4495-b83b-5d60beb03730/manager/0.log" Jul 14 18:56:11.798724 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:11.798689 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6785s5b9_b8af0c89-bb5f-4fea-9f2e-9383b6cc9df4/istio-proxy/0.log" Jul 14 18:56:16.448069 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:16.447950 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nkcpt_f7c8649a-080e-472b-80c7-715d43207da1/migrator/0.log" Jul 14 18:56:16.477243 ip-10-0-129-146 kubenswrapper[2579]: I0714 18:56:16.477205 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nkcpt_f7c8649a-080e-472b-80c7-715d43207da1/graceful-termination/0.log"